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Poor  project  management  and  severe  technical 
problems  have  hindered  National  Weather  Serv¬ 
ice  efforts  to  develop  its  Automation  of  Field 
Operations  and  Services  system.  AFOS  is  at 
least  5  years  behind  schedule,  has  cost  $1Q0 
million -about  $22  million  over  its  original 
budget -and  cannot  meet  its  original  require¬ 
ments. 

To  overcome  AFOS'  deficiencies,  the  Service 
plans  to  design  and  develop  a  new  system  that 
should  meet  all  original  requirements.  It  also 
plans  to  operate  AFOS  "as  is"  while  the  new 
system  is  under  development. 

GAO  is  recommending  that  the  Service  reap¬ 
praise  AFOS  and  subject  it  to  further  testing. 
In  addition,  the  Service  should  prepare  a  com¬ 
prehensive  plan,  including  a  cost-benefit  anal¬ 
ysis,  before  implementing  AFOS  nationwide. 
Because  AFOS,  if  implemented,  would  be  an 
interim  system,  the  Service  should  immediately 
commit  resources  to  developing  a  replacement 
system. 
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To  the  President  of  the  Senate  and  the 
Speaker  of  the  House  of  Representatives 

This  report  describes  major  design  problems  and  management 
deficiencies  with  the  National  Weather  Service's  Automation  of 
Field  Operations  and  Services  (AFOS)  project.  We  conclude  that 
before  NWS  can  proceed  with  national  implementation  it  should 
complete  a  full  economic  assessment  and  thorough  testing  of  all 
aspects  of  the  system  to  determine  whether  national  implementa¬ 
tion  is  cost  effective. 


We  support  the  Service’s  plan  to  develop  a  new  system  to 
replace  AFOS ,  but  the  Service  has  not  adequately  addressed  the 
impact  of  a  new  systems  development  effort  on  its  capacity  to 
run  AFOS  and  the  current  communications  system. 


This  review  was  undertaken  in  response  to  an  April  11 , 
1980,  request  by  Congressman  Willis  D.  Gradison,  Jr.  At  his 
request  we  examined  the  AFOS  project's  justification,  technical 
adequacy,  and  management. 


We  are  sending  copies  of  this  report  to  the  Director, 
Office  of  Management  and  Budget?  the  Administrator,  General 
Services  Administration?  and  other  interested  parties  and  will 
make  copies  available  to  the  public  upon  request. 
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Comptroller  General 
of  the  United  States 
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COMPTROLLER  GENERAL’S 
REPORT  TO  THE  CONGRESS 


PROBLEMS  PLAGUE  NATIONAL 
WEATHER  SERVICE  ADP  SYSTEM 


The  National  Weather  Service  should  halt  imple¬ 
mentation  of  its  automated  data  processing  and 
telecommunications  system  until  it  more  com¬ 
pletely  resolves  the  system's  problems  and 
clearly  establishes  that  the  benefits  of  full 
operation  are  worth  the  substantial  costs. 

The  system  is  called  Automation  of  Field  Opera¬ 
tions  and,  Services  ( AFOS)  .  b  h  r  e  rj"'-’ 

c]  i  S  C  k  5  ■ 


GAO  conducted  this  review  at  the  request  of 
Congressman  Willis  D.  Gradison,  Jr.,  who  was 
concerned  that  implementing  the  system  might 
not  be  practical  or  feasible. 

OVERVIEW  OF  SYSTEM  PROBLEMS 
AND  GAO’S  RECOMMENDATIONS 


After  7  years  of  development  and  expenditures  of 
$100  million,  the  National  Weather  Service  has 
implemented  AFOS  in  two  of  its  four  principal 
regions.  GAO’s  evaluation  of  the  system  found 
substantial  problems  in  its  design,  operation, 
maintenance,  and  management.  GAO  found,  in  addi¬ 
tion,  that  several  of  the  design  problems  are  in¬ 
herent  in  the  system  and  cannot  be  resolved  short 
of  a  complete  redesign. 

Because  of  system  limitations,  the  Weather  Service 
had  to  freeze  the  development  of  AFOS — before 
functions  initially  planned  could  be  added.  As  a 
result,  AFOS  is  not  capable,  for  example,  of  trans¬ 
mitting  radar  imagery  data  to  field  offices  from 
the  radar  systems  currently  being  built.  Also,  it 
cannot  transmit  satellite  imagery  data  to  local 
field  offices.  To  perform  these  and  other  added 
functions,  the  Weather  Service  is  designing  a  to¬ 
tally  new  system  which  it  expects  to  have  in  serv¬ 
ice  by  1989  or  1990. 

Despite  the  limitations  of  AFOS,  the  Weather 
Service  plans  to  complete  national  implementation 
and  to  use  AFOS  on  an  "as  is"  basis  from  1982  to 
1990.  GAO  recommends  that  the  Weather  Service 
proceed  with  national  implementation  only  if 
(1)  thorough  testing  of  all  critical  aspects  of 
the  system  indicates  that  full  operations  are 
feasible  and  (2)  the  Weather  Service  undertakes 
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a  cost/benefit  analysis  concluding  that  full 
implementation  is  cost  effective.  (See  ch .  7.) 

GAO  also  recommends  that  the  Weather  Service: 

— Establish  an  overall  project  management 

office  and  assign  personnel  to  it,  including 
a  project  manager,  on  a  full-time  basis 
both  for  completing  AFOS  and  for  developing 
any  new  system. 

— Adhere  to  standard  software  development  prac¬ 
tices  in  completing  AFOS  and  in  developing 
any  new  system.  (See  ch.  4.) 

— Replace  completely  all  AFOS  software,  hardware, 
and  telecommunications  in  developing  any  new 
system . 

— Contract  out  system  development  activities  when 
they  exceed  in-house  capabilities. 

— Account  for  all  costs,  including  the  full  per¬ 
sonnel  costs  attributable  to  using  AFOS  and 
developing  a  new  system.  (See  ch.  2.) 

Unless  the  Weather  Service  resolves  these  manage¬ 
ment  weaknesses  and  technical  problems,  GAO  be¬ 
lieves  that  AFOS  and  the  effort  to  develop  a  new 
system  face  continued  difficulties  and  a  risk  of 
total  failure. 

SPECIFIC  PROBLEMS 


Originally  planned  to  be  completed  by  1979,  AFOS 
is  currently  not  scheduled  to  operate  until  1984 
without  backup  from  the  system  it  replaces.  Cost 
overruns  approximating  $22  million  have  been  in¬ 
curred  in  the  development  phase.  Expected  cost 
savings  from  automation  have  not  yet  been 
generated.  (See  ch.  2.) 

Delay  and  excess  costs  are  the  result  of  the  follow¬ 
ing  problems: 

— AFOS  hardware  lacks  sufficient  core  memory  to 
accommodate  current  software  or  applications 
initially  planned  for  AFOS;  the  Weather  Service 
cannot  tell  how  much  more  memory  is  needed. 

— AFOS  computer's  operating  system  cannot  meet 
concurrent  processing  requirements  originally 
specified . 
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— Software  subsystems  are  integrated  to  the  extent 
that  modifications  to  one  software  segment  may 
result  in  problems  with  other  segments.  (See 
ch.  4.) 

— Development  of  the  software  and  changes  made 
along  the  way  are  not  sufficiently  documented, 
making  further  development  of  AFOS  difficult 
and  expensive. 

— Reliability  of  a  fully  operational  AFOS  has 
yet  to  be  shown.  (See  ch .  7.) 

— The  "loop"  design  for  AFOS  telecommunications, 
which  connects  the  main  weather  forecast  of¬ 
fices  in  four  continuous  chains,  is  subject 
to  system  problems  whenever  difficulties  are 
encountered  in  any  single  link.  It  requires 
more  uniformity  of  procedures,  system  disci¬ 
pline,  and  centralized  enforcement  than  has 
been  the  practice  in  the  Weather  Service. 

( See  ch .  5 . ) 

— AFOS  cannot  receive  and  process  informati'-'.. 
from  remote  meteorological  observing  locations. 

— The  Weather  Service  has  removed  some  of  the 
system’s  backup  capabilities  to  stabilize  and 
simplify  current  operations. 

— Some  AFOS  hardware  is  already  obsolete  and 
maintenance  is  expensive  because  unavailable 
spare  parts  have  to  be  specially  made.  (See 
ch .  6  . ) 

— AFOS  hardware — computers  and  spare  parts — 
was  procured  prematurely  and  was  not  updated 
as  new  technology  was  made  available  by  the 
vendor  .  ( See  ch  .  6  . ) 

OPTIONS  FOR  FUTURE 

OPERATIONS  AND  COSTS 


The  Weather  Service  is  currently  proceeding  with 
national  implementation  of  AFOS,  which  it  expects 
to  complete  before  the  end  of  1982.  At  the  same 
time  it  plans  to  maintain  the  system  AFOS  re¬ 
places,  an  FAA-owned  set  of  teletype  networks, 
for  backup  purposes,  at  least  until  1984.  GAO 
believes  that  operating  these  two  systems  in  par¬ 
allel  for  3  more  years  is  an  unusually  long  trial 
period  for  AFOS.  Yet,  the  Weather  Service  be¬ 
lieves  it  needs  the  added  security  in  the  event 
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AFOS  fails.  Operating  AFOS  on  this  basis  will 
cost  an  estimated  $144  million  over  the  1982  to 
1990  period.  (See  ch.  2.) 

The  Weather  Service  does  not  view  relying  solely 
on  the  existing  FAA  teletype  system  as  a  viable 
option.  GAO  estimates  that,  if  AFOS  were  to  be 
abandoned,  it  would  cost  the  Weather  Service  only 
$28  million  to  continue  using  the  FAA  system  over 
the  next  8  years.  The  Weather  Service  disputes 
this  estimate,  claiming  that  it  ignores  costs  for 
required  renovations  of  the  FAA  system. 

The  Weather  Service  further  states  that  abandon¬ 
ing  AFOS  and  returning  to  sole  reliance  on  the  FAA 
teletype  system  would  be  impractical  if  not  impos¬ 
sible  because  of  shifts  in  operating  procedures 
which  are  already  in  place. 

GAO  and  the  Weather  Service  agree  that  a  new  system, 
targeted  for  1989  or  1990,  must  be  developed.  GAO 
doubts,  however,  that  the  Weather  Service  has  the 
staff  necessary  to  simultaneously  operate  and  main¬ 
tain  AFOS  and  do  the  work  necessary  to  develop  the 
new  system. 

AGENCY  COMMENTS  AND  GAP's 
OBSERVATIONS  OF  RECENT  TESTS 


GAO  proposed  in  a  draft  version  of  this  report 
provided  to  the  agency  in  June  1981  that  AFOS  be 
abandoned  altogether.  The  Weather  Service  dis¬ 
agreed  and  stated  that  it  planned  a  validation 
test  of  AFOS  in  August-September  1981,  to  demon¬ 
strate  the  readiness  of  AFOS  for  national  imple¬ 
mentation.  The  House  Appropriations  Committee 
asked  GAO  to  observe  these  tests.  GAO  found  that 
some  improvements  have  been  achieved  but  noted 
that  the  tests  did  not  cover  all  operations  that 
AFOS  is  expected  to  perform.  (See  ch.  7.) 

These  recent  improvements  do  not  address  and  were 
not  intended  to  resolve  AFOS'  basic  design  prob¬ 
lems,  such  as  (1)  the  potential  for  major  mal¬ 
functions,  (2)  the  lack  of  computer  memory  capac¬ 
ity,  (3)  the  inflexibility  and  cost  of,  and  the 
level  of  personnel  support  required  by,  the  tele¬ 
communications  system,  and  (4)  the  fact  that  the 
system  cannot  meet  all  of  the  Weather  Service's 
operational  requirements. 

GAO  still  questions  whether  implementing  AFOS  is 
worth  the  cost  in  view  of  its  limited  functional 


capabilities  and  operational  problems.  AFOS  does 
meet  some  of  the  Weather  Service’s  planned  objec¬ 
tives.  It  will  increase  the  information  available 
to  local  weather  offices,  make  information  avail¬ 
able  much  faster,  and  provide  limited  data  proc¬ 
essing  capability  to  field  offices.  In  light  of 
these  benefits,  the  recent  improvements,  and  the 
expenditure  of  $100  million  to  date  on  AFOS,  GAO 
at  this  time  does  not  recommend  outright  abandon¬ 
ment  of  AFOS.  However,  GAO  believes  the  burden 
rests  with  the  Weather  Service  to  show  that  bene¬ 
fits  of  using  AFOS  over  the  next  8  years  exceed 
the  costs,  assuming  that  testing  of  all  essential 
aspects  indicates  that  AFOS  can  reliably  perform 
its  remaining  functions. 
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INTRODUCTION 


BACKGROUND 


The  National  Weather  Service  (NWS)  was  established  in  1970 
as  the  successor  agency  to  the  United  States  Weather  Bureau. 

The  Weather  Service  has  evolved  to  its  present  state  over  more 
than  a  century  of  Government  weather  involvement.  House  Joint 
Resolution  143,  passed  in  February  1870,  authorized  the  Secre¬ 
tary  of  War  to  take  observations  and  warn  of  storms  on  the 
Nation's  waterways.  These  responsibilities  were  transferred  by 
the  Act  of  October  1,  1890  (26  Stat.  653),  to  the  newly  created 
Weather  Bureau,  a  part  of  the  Department  of  Agriculture.  It 
was  transferred  to  the  Department  of  Commerce  by  Reorganization 
Plan  Number  IV  of  1940  (5  F.R.  2421  and  54  Stat.  1236)  and  sub¬ 
sequently  was  made  a  part  of  the  National  Oceanic  and  Atmospheric 
Administration  (NOAA) .  NWS  provides  services  through  its  head¬ 
quarters  in  Silver  Spring,  Maryland;  the  National  Meteorological 
Center  (NMC)  in  Camp  Springs,  Maryland;  and  over  400  offices 
throughout  the  United  States.  Of  its  approximately  5,000  em¬ 
ployees,  1,000  are  in  the  Washington  area  and  4,000  are  in  field 
offices. 

MISSION 

NWS  is  responsible  for  protecting  public  safety,  health, 
and  welfare  and  for  promoting  the  comfort  and  convenience  of  the 
general  public  by  providing  information  on  meteorological  and 
hydrological  conditions.  NWS  is  also  responsible  for  meeting  the 
specialized  information  needs  of  weather- sensitive  segments  of 
the  economy,  such  as  agriculture  and  aviation.  This  mission  is 
accomplished  by  two  types  of  services,  basic  and  specialized. 

Basic  services  include 

— taking  hydrological  and  meteorological  observations, 
analyzing  them,  and  preparing  predictions  of  atmos¬ 
pheric,  hydrologic,  and  marine  conditions  and 

— disseminating  observations,  forecasts,  warnings,  and 
other  information  to  the  public. 

The  public  receives  most  of  this  basic  weather  information 
through  two  services,  the  NOAA  Weather  Wire  and  the  NOAA  Weather 
Radio,  and  through  the  mass  media.  These  services  transmit  weather 
information  to  the  public  and  other  users. 

Specialized  services  in  support  of  specific  needs  include: 

— The  Fire  Weather  Forecast  and  Warning  Services  Programs 
provide  specialized  forecasts,  warnings,  and  consulting 
services  to  Federal,  State,  and  private  fire  management 
interests . 


— The  Marine  Weather  and  Oceanographic  Services  Programs 
provide  for  safety  and  increased  efficiency  on  the  Na¬ 
tion's  waterways. 

— The  Agricultural  Weather  Service  provides  specialized 
services  to  help  increase  farm  production,  improve 
agricultural  efficiency,  conserve  energy,  and  protect 
the  environment. 

— The  Domestic  and  International  Aviation  Weather  Programs 
provide  information  for  safe  and  efficient  flight  opera¬ 
tions  . 

ORGANIZATION 


The  National  Meteorological  Center  is  the  backbone  of  NWS 
operations.  Its  resources  include  large  computer  systems  and 
forecast  models  to  process  weather  information.  Virtually  all 
meteorological  data  collected  arrives  at  NMC  where  it  is  analyzed 
and  processed  into  a  variety  of  forecast  and  guidance  products, 
such  as  temperature  and  barometric  pressure  charts,  that  are 
then  distributed  to  NWS  field  offices,  private  meteorologists, 
the  public  media,  and  governmental  offices.  NMC's  products  cover 
the  entire  globe,  and  the  office  has  been  designated  as  the 
analysis  and  forecast  arm  of  the  World  Meteorological  Center, 
fulfilling  U.S.  global  responsibilities  as  part  of  the  interna¬ 
tional  effort  known  as  the  World  Weather  Watch. 

Other  NWS  offices  also  have  national  weather  forecasting 
responsibilities.  The  National  Severe  Storms  Forecast  Center 
in  Kansas  City,  Missouri,  provides  a  single  source  for  severe 
local  storm  watches.  The  National  Hurricane  Center  serves  the 
same  function  for  hurricane  watches  in  the  Atlantic,  the  Carib¬ 
bean,  and  the  Gulf  of  Mexico,  and  Hurricane  Warning  Centers  in 
Honolulu  and  San  Francisco  provide  this  service  for  the  Pacific. 

NWS  is  organized  into  six  regions — four  covering  the  contig¬ 
uous  United  States,  one  covering  Hawaii  and  the  Pacific  Islands, 
and  one  covering  Alaska.  The  regions  are  designated  as  eastern, 
central,  southern,  western,  Alaskan,  and  Pacific.  The  field 
organization  in  the  regions  is  a  two-tier  system.  At  the  top  are 
52  Weather  Service  Forecast  Offices  (WSFOs)  that  are  responsible 
for  a  specified  geographical  area  usually  corresponding  to  a  State 
boundary.  They  receive  forecast  guidance  and  atmospheric  charts 
from  the  NMC,  refine  them,  and  prepare  forecasts  for  their  areas. 
In  addition,  most  WSFOs  take  observations. 

Each  WSFO  also  supports  from  one  to  eight  smaller  suboffices 
called  Weather  Service  Offices  (WSOs) .  WSOs  constitute  the  second 
tier  of  the  field  system.  Each  WSO  is  connected  to  a  WSFO  and 
is  dependent,  on  it  for  support.  WSOs  receive  selected  NMC  pro¬ 
ducts  plus  the  longer  range  forecast  products  prepared  by  their 
"parent"  WSFOs.  Further,  WSOs  prepare  locally  adapted  weather 
forecasts,  based  on  those  prepared  by  the  parent  WSFO,  which 
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generally  cover  shorter  time  periods  and  smaller  areas.  There 
are  about  235  WSOs  nationwide. 


River  Forecast  Centers  are  an  additional  category  of  field 
offices.  Thirteen  River  Forecast  Centers  nationwide  collect  and 
process  data  and  prepare  forecasts  of  levels  and  flow  rates  along 
river  systems,  water  supply  potential,  and  warnings  of  flood 
conditions. 

Relationship  to  other  Federal  agencies 

The  Federal  Aviation  Administration  ( FAA )  and  two  Department 
of  Defense  services,  the  Navy  and  the  Air  Force,  also  have  respon¬ 
sibilities  for  collecting  and  disseminating  weather  information 
in  the  United  States  and  overseas.  These  three  organizations  and 
NWS,  although  different  in  structure  and  mission,  are  dependent 
on  each  other.  The  basic  weather  mission  of  each  service  involves 
collecting  data,  developing  weather  forecasts,  and  communicating 
that  information  to  users.  Each  of  the  services  relies  on  the 
others  in  meeting  some  of  these  functions  and  for  providing  emer¬ 
gency  backup.  The  exchange  of  information,  observations,  and 
forecasts  among  all  services  is  essential  for  the  operation  of 
each  system. 

FAA,  through  its  network  of  Flight  Service  Stations,  pro¬ 
vides  weather  information  to  the  Nation's  private  and  commercial 
pilots.  Flight  Service  Stations  collect  weather  data  throughout 
the  United  States,  relay  it  to  the  other  services,  and  dissemi¬ 
nate  forecasts  prepared  by  NWS. 

The  Air  Force's  Air  Weather  Service  provides  weather 
information  worldwide  to  U.S.  military  air  and  ground  forces 
tailored  to  meet  their  operational  and  planning  needs.  It  also 
shares  observations  and  forecasts  with  the  other  services. 

The  Navy's  Naval  Oceanographic  Command  meets  the  worldwide 
weather  needs  of  the  U.S.  fleet.  Its  emphasis  is  on  weather 
affecting  the  world's  oceans,  but  it  also  takes  observations 
and  prepares  forecasts  that  are  shared  with  the  other  services. 

CURRENT  COMMUNICATIONS  SYSTEM 


The  communications  network  that  links  NWS  offices  is  a  vast, 
complex,  and  aging  network  comprised  of  about  19  communications 
systems.  Some  of  these  are  operated  by  NWS  and  others  are  the 
responsibility  of  FAA  and  the  Department  of  Defense.  NWS'  cur¬ 
rent  effort  to  automate  its  communications  and  data  handling, 
known  as  the  Automation  of  Field  Operations  and  Services  (AFOS) 
system,  was  planned  to  result  in  savings  by  eliminating  10  of 
these  systems. 

FAA  and  NWS  have  been  sharing  communications  systems  since 
1927.  The  three  primary  teletypewriter  systems  used  by  NWS  for 
transmitting  weather  data  between  offices  are  owned  and  operated 


by  FAA.  These  systems  are  20  to  40  years  old.  Graphic  products 
for  all  the  weather  services  are  transmitted  over  two  facsimile 
networks  owned  and  operated  by  NWS.  FAA  is  developing  its  own 
automated  weather  information  system  and  expects  to  complete  it 
by  1988.  FAA  intends  to  discontinue  its  current  teletypewriter 
service  once  a  replacement  system  is  fully  operational. 

NWS'  DECISION  TO  DEVELOP  AN  ADP 
AND  TELECOMMUNICATIONS  SYSTEM 


In  1973  NWS  decided  to  develop  a  new  communications  and 
information  processing  system.  The  decision  was  based  on  the 
results  of  studies  NWS  commissioned  in  the  late  1960 's  and  early 
1970's  to  determine  the  feasibility  of  introducing  automatic 
data  processing  (ADP)  capabilities  into  field  forecasting  of¬ 
fices.  NWS  felt  that  it  would  be  expected  to  meet  increasing 
demands  for  services  but  that  its  staff  resources  would  not  in¬ 
crease  at  the  same  rate.  (NWS’  staff  level  has  remained  stable 
for  13  years  despite  the  addition  of  new  programs.)  The  studies 
showed  that  a  large-scale  effort  toward  automation  would  be 
required  to  solve  anticipated  personnel  shortages.  Automation 
has  the  capability  to  free  field  personnel  from  time-consuming 
administrative  tasks  associated  with  the  current  communications 
system,  which  is  not  automated,  thus  allowing  more  time  for  pro¬ 
fessional  activities,  such  as  developing  and  using  local  weather 
models  and  historical  weather  analysis. 

NWS  also  believed  that  the  use  of  advancements  in  meteorology 
would  be  accelerated  by  using  computer  analysis  and  prediction 
models  at  the  local  level.  In  addition,  automation  promised  sub¬ 
stantial  improvements  in  response  time  and  in  providing  warnings, 
savings  by  lowering  operating  costs,,  and  reduced  staffing  problems. 

DESCRIPTION  OF  AFOS 

AFOS  is  a  nationwide  telecommunications  network  of  mini¬ 
computers  designed  to  connect  most  NWS  offices.  The  system 
will  allow  each  field  office  to  send  and  receive  information 
needed  to  meet  forecasting  and  observation  responsibilities 
and  provide  local  computer  processing  capability.  AFOS,  as 
designed,  would  work  as  follows. 

Local  meteorological  observations  (wind,  temperature,  rain¬ 
fall,  barometric  pressure,  etc.)  are  entered  into  the  system  by 
NWS  field  offices  and  transmitted  to  the  NMC.  The  NMC  takes 
this  raw  information  (the  local  meteorological  observations)  and 
uses  computer  models  to  prepare  detailed  guidance  and  weather 
projections  (forecasts,  maps,  etc.).  The  NMC  then  sends  these 
products  over  the  telecommunications  system  to  the  field  offices. 

The  local  offices  receive  and  store  the  NMC  products  for  use 
in  preparing  local  forecasts.  The  field  forecaster,  using  both 
graphic  and  alphanumeric  display  terminals,  prepares  the  local 
weather  forecast  by  tailoring  the  NMC's  general  guidance  and  the 
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more  specific  forecasts  of  the  WSFOs  to  local  conditions.  AFOS 
would  permit  improved  analysis  by  overlaying  graphically  presented 
data  on  a  display  screen.  This  allows  a  forecaster  to  use  more 
information  and  a  greater  number  of  information  sources  in  pre¬ 
paring  forecasts.  In  addition,  AFOS  would  provide  the  information 
in  a  more  usable  form  and  would  add  the  capability  of  local  ADP 
processing.  Local  ADP  processing  would  consist  of  running  local 
weather  models  similar  to  those  used  at  the  NMC  to  tailor  the 
large  models  to  local  requirements. 

AFOS  requires  a  central  office  responsible  for  assessing 
system  status,  assisting  in  recovery  from  failures,  maintaining 
system  software,  and  driving  the  system.  The  System  Monitoring 
and  Coordination  Center  (SMCC)  has  been  established  for  such 
purposes . 

The  telecommunications  system  is  designed  to  move  infor¬ 
mation  from  the  field  to  the  NMC  and  back  to  the  field.  It  is 
currently  designed  to  be  compatible  with  NWS'  organizational 
structure.  WSFOs  in  each  of  NWS'  four  contiguous  regions  are 
connected  in  a  single  continuous  circuit  and  information  passes 
in  both  directions  around  this  circuit  to  every  WSFO.  This  type 
of  telecommunications  design  is  called  a  loop.  All  four  regional 
loops  are  connected  at  the  SMCC,  as  is  the  NMC.  WSOs  are  connected 
to  their  "parent"  WSFOs  by  direct  communication  lines  similar  to 
spokes  in  a  wheel.  This  type  of  communications  design  is  called 
a  star.  The  Alaskan  and  Pacific  regions  are  connected  to  the 
SMCC  by  spur  links. 

Each  WSFO  has  two  minicomputers  so  that  one  will  remain 
operational  if  the  other  fails.  The  larger  WSOs  will  have  one 
minicomputer.  The  rest  of  the  field  offices  will  have  a  terminal 
connected  to  a  WSFO  which  will  allow  them  to  enter  and  receive 
information  from  the  system.  They  will  not  be  able  to  process 
data  locally.  NWS  has  not  procured  or  rented  the  terminals  for 
these  offices  and  does  not  plan  to  do  so  until  AFOS  is  fully 
operational . 

The  AFOS  hardware  configuration  varies  depending  on  the 
type  and  size  of  the  field  office  in  which  it  is  installed.  In 
addition  to  the  computer,  each  field  office  has  a  control  console 
and  a  number  of  work  stations.  The  work  stations  consist  of 
graphic  and  alphanumeric  display  terminals  which  look  and  func¬ 
tion  like  TV  screens.  A  forecaster  can  put  maps  containing 
weather  information  on  the  graphic  screen  and  overlay  additional 
maps  with  different  types  of  information.  Alphanumeric  terminals 
are  used  for  message  composition,  to  input  information  into  the 
system,  and  to  display  information. 

OBJECTIVES,  SCOPE,  AND  METHODOLOGY 

Our  review  was  performed  at  the  request  of  Congressman 
Willis  D.  Gradison,  Jr.,  to  determine  the  problems  that  have 
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caused  delays  in  implementing  the  AFOS  system.  Our  review  in¬ 
cluded  management  issues  such  as  system  planning,  costs  and  ben¬ 
efits,  coordination  with  external  users,  management  adequacy, 
and  technical  issues  related  to  ADP  system  development  from  sys¬ 
tem  design  through  testing.  We  extensively  studied: 

— NWS'  requirements  both  now  and  in  the  future,  and  AFOS' 
intended  impact  on  service  to  the  public. 

— The  basis  on  which  AFOS  was  justified — the  use  of  a 
distributed  data  base  management  and  communications 
system  1/  to  meet  NWS'  needs. 

— NWS'  organizational  structure  and  its  impact  on  ADP  sys¬ 
tems  development,  including  the  role  of  the  NWS  field 
organization  in  developing  and  implementing  AFOS. 

— Management's  control  of  the  system's  development  and  its 
capability  to  develop  an  ADP  system. 

— The  system's  software,  with  emphasis  on  its  design,  to  de¬ 
termine  if  it  could  effectively  accommodate  changing  needs. 

— The  adequacy  of  the  telecommunications,  hardware,  and 
logistics  systems. 

— The  validation/demonstration  test  plan  for  the  AFOS 
August-September  1981  test. 

NWS  staffs  worked  very  closely  with  our  staff  during  the 
study.  We  freely  exchanged  information  on  a  regular  basis, 
including  information  on  tentative  conclusions  and  recommendations. 
This  approach  was  encouraged  by  Congressman  Gradison. 

Our  methodology  took  into  account  the  operational  philos¬ 
ophy  of  NWS.  Because  NWS  has  a  decentralized  management  struc¬ 
ture,  each  region  developed  independent  management  plans  for  AFOS 
operation  and  implementation.  We  addressed  these  differing  ef¬ 
forts  and  activities  by  performing  indepth  work  in  two  regions 
and  limited  field  work  in  two  others.  This  work  included  discus¬ 
sions  with  senior  officials  and  forecasters  at  local  field  offices 
in  each  region  and  extensive  onsite  observations.  We  reviewed  NWS 
directives  on  the  duties  of  field  offices  and  the  methods  by  which 
they  are  carried  out.  In  addition,  we  reviewed  the  needs  of  three 
separate  groups  of  users  within  each  region:  WSFOs,  WSOs,  and 
River  Forecast  Offices.  Each  has  different  requirements  and  end 
users  of  its  services. 


1/NWS’  distributed  data  base  management  system  includes  the 
distribution  of  a  data  base  at  remote  locations  and  the  use 
of  programmable  minicomputers  to  process  data  in  200  field 
offices. 
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We  performed  detailed  work  in  the  central  and  western  regions 
where  the  development  of  AFOS  is  the  most  advanced.  Our  work  in 
the  remaining  regions  provided  sufficient  information  to  address 
field  operations  as  a  whole. 

In  addition,  we  performed  limited  work  at 

— the  NWS  Technical  Training  Center  and  Repair  Depot  in 
Kansas  City,  Missouri,  on  AFOS'  logistics  and  training 
program; 

— the  FAA  Weather  Message  Switching  Center  in  Kansas  City, 
Missouri,  on  NWS'  current  system,  which  is  primarily  run 
by  FAA; 

— the  Air  Weather  Service  at  Offutt  Air  Force  Base,  Omaha, 
Nebraska,  on  NWS'  backup  operational  support  in  the  event 
of  a  major  disaster  to  the  NMC ; 

— the  Ford  Aerospace  Communications  Corporation,  the  con¬ 
tractor  which  assembled  the  AFOS  hardware,  in  Palo  Alto, 
California,  on  the  performance  characteristics  of  the 
AFOS  system  and  future  logistics  costs  and  support; 

— the  Data  General  Corporation,  the  contractor  which  supplied 
the  AFOS  minicomputers,  in  Westboro,  Massachusetts;  and 

— the  Environmental  Research  Laboratory,  the  primary  source 
of  research  for  U.S.  meteorological  services,  in  Boulder, 
Colorado,  on  new  techniques  and  services  that  would  need 
AFOS  support. 

In  addition,  we  met  with  the  major  contractors  who  have 
supported  NWS  in  developing  AFOS,  to  obtain  information  on 
studies  prepared  to  justify  AFOS  and  to  gain  a  better  understand¬ 
ing  of  how  the  system  performs  and  is  planned  to  perform.  We  also 
interviewed  key  personnel  and  reviewed  pertinent  documents  at  all 
major  NWS  headquarters  support  offices.  The  Director  of  NWS  and 
the  Director  of  our  Community  and  Economic  Development  Division 
corresponded  during  the  final  phase  of  our  review.  This  exchange 
addressed  our  concern  that  management  and  technical  deficiencies 
were  impediments  to  implementing  AFOS.  The  Director  of  NWS  stated 
that  improvements  recently  made  support  NWS'  position  that  AFOS 
is  working  satisfactorily  and  that  it  should  be  implemented.  After 
providing  NWS  with  a  copy  of  our  draft  report  for  official  review 
and  comment,  we  continued  to  perform  onsite  audit  work.  This  ad¬ 
ditional  work,  performed  in  response  to  a  request  contained  in  a 
report  of  the  House  Committee  on  Appropriations,  1/  consisted  of 
observing  NWS'  August-September  1981  validation  test  of  AFOS  at 


1/House  Appropriations  Committee  Report  97-180,  July  16,  1981, 
p.  14 . 


selected  sites  in  the  western  and  central  regions.  We  also 
reviewed  the  preliminary  results  of  the  validation  test  and  dis¬ 
cussed  our  observations  with  the  Director  of  NWS,  the  AFOS  project 
manager,  and  other  key  officials.  We  met  with  officials  of  the 
National  Oceanic  and  Atmospheric  Administration  and  the  Department 
of  Commerce  and  obtained  information  on  their  involvement  in  ap¬ 
proving  AFOS  and  concerns  over  the  project's  development. 


CHAPTER  2 


STATUS  AND  COST  OF  AFOS 


NWS  began  planning  for  the  AFOS  system  in  the  1960's  and 
began  developing  it  in  1974.  AFOS  was  to  be  fully  operational  by 
August  1979  at  a  cost  of  $77.6  million.  However,  the  project 
has  been  delayed  by  severe  technical  problems  with  hardware,  soft¬ 
ware,  and  telecommunications,  in  addition,  for  various  reasons 
AFOS  will  not  realize  expected  cost  savings.  AFOS  development 
costs  have  been  understated  because  NWS  did  not  use  standard  ac¬ 
counting  procedures  and  controls.  By  our  best  estimate,  as  of 
September  30,  1980,  NWS  had  spent  $100  million  on  the  project, 
and  the  earliest  that  AFOS  could  be  fully  operational  is  1984. 

Even  if  AFOS  were  fully  implemented  by  1984,  the  system 
planned  to  be  made  operational  is  not  the  AFOS  originally 
designed  but  a  system  with  limited  capability.  NWS  indicates 
that  it  plans  to  design  and  develop  a  new  system  that  will  meet 
original  requirements.  This  system  is  estimated  to  be  completed 
in  1989  at  a  cost  of  $125  to  $150  million.  If,  as  planned,  NWS 
uses  AFOS  on  an  interim  basis  until  the  new  system  is  completed 
in  1989,  AFOS  could  cost  an  additional  $144  million. 

CURRENT  STATUS  OF  AFOS 

AFOS  has  been  phased  into  actual  field  operations  over  the 
past  several  months  and  is  now  being  used  as  a  primary  tool  by 
forecasters  in  the  preparation  and  delivery  of  weather  services 
in  selected  field  offices.  Currently,  the  sites  using  AFOS  are 
located  in  two  of  the  four  contiguous  NWS  regions — western  and 
central.  By  the  end  of  fiscal  year  1982,  NWS  anticipates  full 
implementation  of  AFOS  at  all  planned  sites  in  all  four  contiguous 
regions.  NWS  plans  to  continue  using  the  current  communications 
system  operated  by  FAA  through  the  end  of  fiscal  year  1984. 

All  AFOS  hardware,  software,  and  telecommunications  are  in¬ 
stalled  with  all  four  regional  loops  being  driven  by  the  System 
Monitoring  and  Coordination  Center.  Based  on  the  validation/ 
demonstration  test  that  was  conducted  in  August-September  1981, 

NWS  believes  that  the  performance  of  AFOS  will  continue  to  improve 
as  field  personnel  gain  experience  and  system  deficiencies  are 
corrected . 

Despite  these  improvements,  AFOS  still  has  problems  with 
software,  hardware,  and  telecommunications  that  prevent  it  from 
performing  as  originally  planned  and  that  make  future  development 
and  enhancement  difficult.  In  early  1981  NWS  conducted  an  opera¬ 
tional  test  to  assess  these  problems.  Following  this  test  the 


Director  of  NWS  stated  in  a  memorandum  1/  that  solutions  to  prob¬ 
lems  affecting  future  development  and  enhancements  of  the  system, 
such  as  integrating  radar  and  satellite  data,  were  not  possible 
without  a  major  redevelopment  effort,  including  replacing  the 
software,  hardware,  and  telecommunications  systems.  The  Director 
further  stated  that  the  addition  of  any  further  capabilities 
needed  to  support  current  operations  is  not  possible,  and  in 
fact  some  capabilities  already  in  AFOS  would  have  to  be  removed  in 
order  to  stabilize  it.  For  example,  some  of  the  disaster  protec¬ 
tion  and  system  recovery  features  have  been  removed. 

Other  problems  were  disclosed  in  the  memorandum  that  will 
prevent  the  system  from  operating  reliably.  Examples  of  these 
include  fault  isolation  and  identification,  the  ability  to  monitor 
the  status  of  the  network,  recovery  from  degraded  modes  of  opera¬ 
tion,  message  composition,  and  the  ability  to  send  messages  through 
the  NOAA  Weather  Wire.  The  effect  of  these  problems  is  a  reduction 
in  the  quality  of  service  NWS  can  provide,  greater  risk  of  field 
offices  not  being  operational,  and  an  increase  in  AFOS  operating 
costs.  AFOS  development  continues  in  an  effort  to  correct  these 
and  other  deficiencies. 

In  August  and  September  1981,  NWS  conducted  a  second  major 
operational  test  on  the  basis  of  which  NWS  was  to  decide  whether 
to  continue  developing  AFOS  or  abandon  it.  The  criteria  against 
which  AFOS  was  judged  in  this  test  were  limited.  (See  ch.  7  for 
further  discussion  of  test  criteria.) 

Regardless  of  the  final  results  of  the  operational  test, 
still  being  assessed  at  this  date,  NWS  plans  to  develop  a  new 
system  that  should  overcome  the  deficiencies  of  AFOS  and  meet 
future  needs.  Although  planning  for  this  system  has  just  begun, 
the  Director  of  NWS  estimates  that  it  could  be  operational  by 
1989  and  that  its  costs  of  $125  to  $150  million  would  be  com¬ 
parable  to  a  completed  AFOS  system. 

ACCOUNTING  PROCEDURES 


Some  costs  that  should  have  been  attributed  to  the  AFOS 
program  have  not  been  included  in  NWS'  AFOS  cost  figures.  We 
examined  NWS'  AFOS  cost  data  and  determined  that  AFOS  program 
costs  have  been  underreported  by  about  $18  million.  Further, 
costs  have  not  been  classified  or  presented  to  management  in  a 
meaningful  and  useful  format.  These  shortcomings  occurred  because 
NWS  did  not  use  project  cost  accounting  techniques  that  require 
detailed  cost  identification  data  and  did  not  establish  reporting 
or  control  mechanisms.  Although  we  informed  NWS  of  these  defi¬ 
ciencies,  it  still  has  not  established  a  good  accounting  system 


1/National  Weather  Service  Director's  memorandum  to  NWS  regional 
and  office  directors  regarding  "AFOS  National  Implementation 
Decision,"  March  19,  1981. 
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for  AFOS.  Moreover,  future  AFOS  development  costs  will  continue 
to  be  understated. 

Two  primary  documents  cover  accounting  procedures  required 
for  major  systems.  These  are  Office  of  Management  and  Budget  (OMB) 
Circular  A-109,  "Major  System  Acquisitions,"  and  Federal  Govern¬ 
ment  Accounting  Pamphlet  Number  4,  "Guidelines  for  Accounting  for 
Automatic  Data  Processing  Costs."  Both  documents  point  out  the 
necessity  to  identify  significant  elements  of  costs  directly  re¬ 
lated  to  acquiring  systems  so  that  informed  management  decisions 
may  be  made  at  each  phase  in  the  entire  life  cycle  of  a  system. 
These  decisions  include  the  most  economical  time  to  replace  a 
system  and  alternatives  in  acquiring  new  systems.  In  summary, 
these  documents  state  that  accumulating  the  purchase  cost  of  the 
system's  components  is  only  one  step  in  determining  the  system's 
total  cost.  An  agency  must  further  determine  the  costs  of  all 
resources,  including  personnel  involved  in  developing,  procuring, 
installing,  and  testing  the  system,  including  the  system's  soft¬ 
ware.  A  system  must  also  be  assigned  its  share  of  the  rent  or 
lease  expense  of  buildings,  utilities,  and  management  overhead. 
Additionally,  training  costs  are  an  integral  part  of  a  system's 
development  expense. 

Life-cycle  costing  as  required  by  OMB  Circular  A-109  re¬ 
quires  an  agency  to  accumulate  all  direct,  indirect,  and  recur¬ 
ring  costs  of  a  system  for  each  phase  of  its  life  span.  Under 
this  practice  a  system  remains  in  its  development  phase  until 
it  can  perform  the  function  for  which  it  is  intended. 

The  cost  of  personnel  is  the  primary  area  in  which  AFOS 
development  costs  were  not  collected.  For  example,  training 
costs  included  only  the  cost  of  providing  the  training,  and 
not  the  higher  salaries  and  travel  costs  of  personnel  receiving 
the  training.  Also,  personnel  costs  for  development,  testing, 
and  management  have  not  been  reported.  Because  NWS'  develop¬ 
ment  approach  to  AFOS  included  many  people  actively  involved 
with  the  AFOS  project,  the  magnitude  of  this  personnel  cost 
problem  is  significant.  For  example,  the  western  region  re¬ 
ported  that  in  one  5-month  period,  over  3,500  hours  had  been 
devoted  to  AFOS  that  were  not  charged  to  the  AFOS  project. 

Overhead  is  another  area  in  which  costs  were  not  applied 
to  AFOS.  A  proportionate  share  of  an  agency's  management  and 
administrative  expense  should  be  applied  to  a  project's  costs 
in  addition  to  overhead  expenses  such  as  utilities  and  rent. 

These  are  significant  expenses  for  a  comprehensive  ADP  system 
such  as  AFOS. 

NWS  has  not  accounted  for  other  AFOS  development  expenses 
by  transferring  to  operational  accounts  all  maintenance  and 
parts  expense  for  the  AFOS  hardware.  During  installation  and 
testing  these  expenses  have  been  considerable.  Further,  because 
AFOS  is  not  an  operational  system,  these  expenses  should  be 
considered  developmental. 
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NWS  accounting  procedures  did  not  accurately  allocate  or 
record  expenditures  in  the  following  cost  categories: 

— Increased  utilities  (primarily  electricity)  and  space 
required  to  support  AFOS. 

— Additional  air  conditioning  installed  at  field  sites  to 
meet  regional  AFOS  environmental  conditions. 

— New  facilities  required  for  NWS  field  offices,  in  part 
due  to  AFOS,  and  modifications  to  facilities  required 
to  support  AFOS  hardware. 

— Management  overhead  at  the  field  and  headquarters  level 
which  was  not  applied  to  AFOS  development  costs.  For 
example,  the  salary  of  the  AFOS  project  manager  was  not 
charged  to  AFOS  over  the  last  7  years  of  development. 

NWS  has  not  corrected 
accounting  deficiencTes 

In  November  1980  we  informed  the  Director  of  NWS  of  the  defi 
ciencies  in  project  accounting.  However,  NWS  still  does  not  in¬ 
clude  overhead  and  staff  costs  in  computing  future  AFOS  operating 
and  development  costs.  For  this  reason  operating  costs  are  under 
stated  by  approximately  $2.1  million  annually.  As  a  result,  AFOS 
operating  costs  from  fiscal  years  1981  through  1989  will  be  about 
$18.9  million  higher  than  reported  by  NWS. 

Essentially,  NWS  regards  personnel  costs  as  a  fixed  expense, 
even  though  Government  policy  regards  them  as  a  variable  expense. 
The  cost  of  AFOS  personnel  to  NWS  is  lost  to  other  projects  to 
which  their  time  could  have  been  devoted.  The  Director  of  NWS 
indicated  that  it  was  unnecessary  to  charge  these  costs  to  AFOS 
because  NWS  would  incur  them  with  or  without  AFOS.  Federal  ac¬ 
counting  regulations,  however,  require  that  personnel  costs  be 
assigned  to  the  project. 

AFOS  DEVELOPMENT  COSTS  TO  DATE 


A  partial  accounting  of  AFOS  development  costs  was  available 
in  the  NWS  accounting  system,  and  NWS  at  our  request  provided 
a  list  of  estimated  unaccounted  costs  in  identified  areas.  These 
figures  place  AFOS  development  costs  through  fiscal  year  1980  at 
about  $100  million.  According  to  NWS,  it  received  $77.6  million 
in  congressional  appropriations  specifically  for  AFOS;  $5.9  mil¬ 
lion  in  additional  NWS  resources  acknowledged  by  NWS;  and  $18  mil 
lion,  primarily  for  personnel,  that  has  not  been  recorded  by  the 
NWS  accounting  system.  The  misallocation  of  these  expenses  to 
other  NWS  programs  occurred  primarily  because  NWS  has  not  exer¬ 
cised  management  controls  over  AFOS  development  and  costs. 

Our  concern  is  the  absence  of  cost  information  for  NWS  and 
the  Congress  to  use  in  managing  the  project  as  the  funds  are 
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budgeted  and  expended.  If  NWS  is  to  have  needed  information  for 
managing  the  program  in  the  future,  complete  AFOS  cost  collection 
procedures  are  required.  This  is  particularly  relevant  in  light 
of  the  additional  $144  million  NWS  plans  to  spend  on  AFOS  and  a 
potential  $125  to  $150  million  it  plans  to  spend  on  developing  a 
new  system.  We  identified  expenditures  of  about  $18  million  that 
were  unaccounted  for  in  the  areas  of  overhead,  training,  and  some 
personnel  costs. 


GAO  Estimates  of  AFOS  Costs 
for  Fiscal  Years  1974-80  (note  a) 


Hardware 
Software 
Training 
Communications 
Uncollected  costs 

(primarily  personnel) 
Expenditures,  fiscal  1974 
and  fiscal  1975  (note  b) 


$  64,038,250 
12,744,150 
1,525,900 
871,900 

18,000,000 

2,980,000 


Total 


$100,160,200 


a/These  figures  are  rough  approximations. 


b/The  AFOS  project  did  not  appear  as  a 
budget  item  until  fiscal  year  1976  and 
costs  were  not  separated  before  that 
time . 


Current  NWS  accounting  procedures  do  not  provide  management 
with  information  and  assurance  that  funds  are  spent  effectively. 
Further,  no  system  exists  to  compare  allocated  funds  with  actual 
expenditures.  Once  AFOS  funds  are  allocated,  actual  expenditures 
are  not  tracked.  Moreover,  existing  accounting  procedures  do  not 
provide  for  establishing  accounts  for  capital  equipment  costs. 
Software  costs  are  not  separated  from  other  costs,  and  hardware 
costs  include  only  procurement  and  not  installation  or  testing. 

AFOS  COST  SAVINGS 
WILL  NOT  BE  REALIZED 


^cOS  was  originally  planned  and  presented  as  a  system  that 
would,  -.ong  other  objectives,  pay  for  itself  through  increased 
efficiencies.  These  savings  were  to  result  from  eliminated  or 
avoided  positions,  removal  of  the  current  communications  system, 
and  greater  employee  productivity.  The  savings  have  been  delayed 
and  the  efficiencies,  in  our  opinion,  will  not  occur.  Further, 
AFOS  will  not  pay  for  itself  through  savings  but  will  require 
at  least  $15  million  annually  for  operating  costs  as  opposed  to 
about  $4  million  needed  annually  to  operate  the  current  communi¬ 
cations  system. 
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NWS  estimates  that  AFOS'  annual  operational  costs  will  be 
$13  million  and  that  $2.6  million  in  savings  can  be  achieved  by 
removing  part  of  the  current  system.  We  determined  that  the  NWS 
estimate  does  not  include  $2  million  in  annual  overhead  cost, 
bringing  our  estimate  of  AFOS'  operational  costs  to  $15  million 
annually.  We  also  believe  NWS  cannot  remove  the  current  communi¬ 
cations  system  and  will  not  achieve  the  $2.6  million  in  annual 
savings  it  has  estimated  for  this  change. 

The  current  system  which  AFOS  is  to  replace  is  operated  pri¬ 
marily  by  FAA ;  NWS  pays  only  for  direct  support  expenditures 
and  the  equipment  it  connects  into  the  system.  As  NWS  implements 
AFOS,  it  must  not  only  support  a  vastly  more  complex  system  at 
the  field  level  but  must  also  assume  the  cost  of  system  develop¬ 
ment,  management,  and  maintenance — functions  currently  provided 
at  no  cost  by  FAA. 

Over  a  7-year  development  period  the  AFOS  system  has  changed 
in  many  ways.  Changes  in  basic  operating  requirements  for  the 
system  have  been  caused  by  operational  and  design  considerations. 
The  current  communications  system  AFOS  was  to  replace  in  1979 
will  now  remain  in  operation  as  a  backup  through  at  least  1984. 

NWS  staffing  levels  have  also  changed,  and  many  positions  that 
AFOS  was  to  eliminate  have  already  been  removed  by  budget  cuts. 

NWS  now  recognizes  that  AFOS  will  not  produce  cost  savings  equiv¬ 
alent  to  or  exceeding  its  development  and  annual  operating  costs. 

At  the  same  time,  the  AFOS  system  as  currently  designed  will  pro¬ 
vide  new  services  and  greater  capabilities  than  the  existing  sys¬ 
tem.  However,  NWS  has  yet  to  perform  a  full  cost-benefit  analysis 
to  compare  the  capabilities  and  benefits  of  AFOS,  as  it  is  designed 
today,  against  the  total  costs  of  continuing  to  rely  on  the  current 
communications  system  without  AFOS. 

POTENTIAL  COSTS  TO  COMPLETE  AFOS 

NWS  plans  to  develop  a  new  system  to  meet  future  needs  and 
overcome  the  deficiencies  of  AFOS.  NWS  intends  to  operate  AFOS, 
despite  its  deficiencies,  while  the  new  system  is  being  developed. 
The  future  cost  of  the  AFOS  system  shown  below  is  based  on  data 
provided  by  NWS. 


Future  Cost  of  AFOS 
Fiscal  Years  1982-89 


(000,000  omitted) 


Operate  present  FAA  communications 

system  (note  a)  $  6 

Complete  AFOS  and  operate  through  1989  138 

Develop  new  system  125 

Total  $269 


a/NWS  plans  to  remove  the  current  communicatio'.is  system  in  1984. 
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AFOS  will  not  generate  the  cost  savings  originally  envisioned 
but  will  incur  $15  million  in  additional  annual  operating  expenses 
The  AFOS  project  is  5  years  behind  schedule;  has  cost  $100  million 
$22  million  over  budget;  and  has  fewer  capabilities  than  its  origi 
nal  design.  Yet  AFOS  still  falls  short  of  meeting  needed  require¬ 
ments.  Therefore,  NWS  needs  a  new  system  that  can  perform  many  of 
the  functions  for  which  AFOS  was  originally  designed. 

NWS'  current  communications  system  is  scheduled  to  remain  in 
place  to  back  up  AFOS  until  at  least  1984. 

NWS  did  not  follow  established  Government  regulations  and 
guidance  in  accounting  for  AFOS  costs.  As  a  result,  NWS  does  not 
take  into  account  the  true  cost  of  AFOS  development.  Further,  by 
not  following  these  regulations,  NWS  is  understating  future 
operating  costs  by  $2  million  annually. 

RECOMMENDATIONS 

We  recommend  that  the  Secretary  of  Commerce  direct  NWS  to 

— account  for  all  AFOS  costs,  including  the  full  personnel 
costs  attributable  to  developing  and  using  AFOS,  and 

— follow  accounting  regulations  prescribed  in  OMB  Circular 
A-109  in  accounting  for  system  development  costs,  includ¬ 
ing  life-cycle  costs. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

NWS  disagreed  with  our  conclusions  that  the  AFOS  project  is 
5  years  behind  schedule  and  that  its  development  stage  will  be 
completed  when  NWS  has  fully  implemented  AFOS  and  removed  the 
current  telecommunications  system  run  by  FAA.  NWS  has  estimated 
that  this  process  will  be  completed  in  1984,  5  years  after  its 
original  estimate  of  1979.  NWS'  position  is  that  AFOS'  develop¬ 
ment  will  be  complete  when  all  scheduled  offices  are  using  AFOS, 
which  it  expects  to  occur  in  1982. 

AFOS  will  be  fully  operational,  in  our  view,  only  when  the 
current  system  is  removed  in  1984  and  all  stations  are  using 
AFOS.  In  its  internal  plans  developed  before  responding  to  our 
report,  NWS  considered  removal  of  the  current  system  as  the  end 
of  the  development  phase,  just  as  we  do.  Furthermore,  in  the 
private  sector  and  in  Government,  the  time  of  removal  of  previous 
systems  is  an  accepted  management  practice  for  indicating  the  end 
of  a  development  phase.  By  this  measure,  AFOS  is  5  years  behind 
schedule . 

NWS  stated  that  we  had  overstated  AFOS  development  costs 
by  at  least  $11  million.  NWS  attributed  this  amount  to  (1)  a 


conceptual  difference  in  applying  overhead  and  (2)  differences 
in  applying  costs  for  personnel  involved  in  routine  support 
functions.  NWS  further  maintained  that  planned  AFOS  spending 
has  been  based  all  along  on  using  personnel  funds  in  conjunc¬ 
tion  with  other  reprogramed  development  funds. 

NWS  maintains  that  its  application  of  overhead  is  consis¬ 
tent  with  the  NOAA  financial  system,  which  GAO  approved.  It  is 
our  contention  that  NWS  did  not  use  the  GAO-approved  accounting 
system  and  its  method  of  applying  overhead  for  the  AFOS  project. 
There  is  no  evidence  showing  that  the  correct  overhead  was  ap¬ 
plied  to  the  future  AFOS  program  cost  estimates.  Also,  the  NWS 
resource  management  staff  confirmed  that  overhead  was  not  ap¬ 
plied  to  AFOS  costs. 

A  point  of  difference  between  NWS  and  us  is  the  assignment 
of  costs  for  all  AFOS  developmental  staff.  NWS  attributes  many 
of  these  personnel  costs  to  "routine  cross  utilization  of 
support."  The  magnitude  of  NWS'  "cross  utilization"  is  large, 
amounting  to  tens  of  thousands  of  hours.  Further,  the  work 
performed  by  these  personnel  was  the  direct  development,  main¬ 
tenance,  and  testing  of  the  AFOS  system.  It  should  therefore 
be  charged  to  the  project.  Further,  in  the  1976  Program  De¬ 
velopment  Plan,  NWS  specified  the  resource  requirements  for 
AFOS  and  did  not  include  the  use  of  existing  funding  or  repro¬ 
gramed  funds.  In  its  response  NWS  stated  that  it  intends  to 
begin  charging  appropriate  direct  labor  costs  to  the  AFOS  pro¬ 
gram,  including  the  project  manager's  salary. 

NWS  stated  that  it  is  not  necessary,  as  we  recommend,  that 
it  follow  accounting  procedures  prescribed  in  OMB  Circular  A-109. 
It  stated  that  the  Department  of  Commerce  did  not  implement 
A-109  until  1978  and  that  "at  that  point  the  acquisition  of  the 
AFOS  system  was  so  nearly  complete  that  it  was  not  considered 
to  be  applicable."  While  it  is  true  that  the  AFOS  hardware  was 
purchased  by  1978,  approximately  $60  million  has  been  spent  since 
then  on  continuing  development  efforts. 

OMB  Circular  A-109  covers  costs  beyond  the  purchase  of 
components.  Its  approach  is  based  on  the  principle  that  an  ac¬ 
curate  picture  of  system  acquisition  costs  can  be  gained  only 
by  viewing  a  system's  entire  life-cycle  costs.  Life-cycle  costs 
would  therefore  include  testing,  operating,  and  supporting  the 
AFOS  system.  The  requirements  of  A-109  should  have  been  followed. 

NWS  also  stated  that  the  recommendation  to  follow  OMB  A-109 
is  unnecessary  because  "NWS  will,  as  it  has  in  the  past,  comply 
with  Department  of  Commerce's  accounting  system."  We  believe 
our  recommendation  is  still  valid.  As  we  note  in  the  report, 

NWS'  application  of  overhead  and  personnel  costs  to  AFOS  has  not 
been  in  accordance  with  the  Department's  or  NWS'  own  procedures. 
Further,  as  NWS  noted  in  its  response,  it  has  been  required  by 


the  Department  of  Commerce  to  use  OMB  A-109  since  1978.  This 
requirement  is  still  not  being  met. 

NWS  also  objected  to  our  estimate  that  developing  a  new 
system  would  cost  $125  to  $150  million.  We  based  our  estimate 
on  AFOS  development  costs  which,  when  completed,  will  reach 
$150  million  or  more.  This  is  the  only  historical  basis  for 
an  estimate.  With  the  effects  of  inflation  on  personnel  costs 
for  software  development  (the  major  costs  of  an  ADP  system), 
it  is  very  unlikely  that  a  now  system  would  cost  less  than  $125 
to  $150  million.  Our  estimate  is  also  consistent  with  an  NWS' 
estimate  on  March  19,  1981,  of  the  approximate  cost  of  a  new 
system. 

Although  additional  funds  may  be  required  to  maintain  the 
current  communications  system,  continuing  to  operate  AFOS  would 
be  considerably  more  expensive.  This  is  especially  true  in  view 
of  NWS'  March  1981  technical  assessment  which  states  that  be¬ 
tween  $12  and  $15  million  will  be  required  to  upgrade  AFOS  to  a 
minimum  level  of  acceptability.  This  does  not  include  any  spend¬ 
ing  to  resolve  basic  AFOS  design  deficiencies  and  constraints. 

NWS  stated  that  we  did  not  offset  the  benefits  of  AFOS 
against  the  cost  savings  from  not  operating  the  system.  During 
our  review  of  AFOS  we  made  clear  our  view  that  NWS  should  pre¬ 
pare  a  cost-benefit  analysis  of  AFOS.  Yet,  NWS  stated  that  this 
was  unnecessary  and  that  it  had  no  plans  to  conduct  such  an  analy¬ 
sis.  Without  the  needed  benefit  and  cost  information,  neither 
NWS  nor  any  other  reviewing  organization,  including  NOAA ,  cogni¬ 
zant  congressional  committees,  or  our  staff,  can  offset  benefits 
against  costs.  However,  we  did  find  that  the  cost  savings  that 
NWS  originally  identified  as  AFOS  benefits  will  not  be  realized. 

NWS  stated  that  the  cost  to  operate  and  maintain  AFOS  is 
close  to  the  original  projections  made  in  1976.  Since  we  did 
not  review  this  aspect,  we  have  no  basis  to  accept  or  reject  NWS' 
statement.  Given  the  lack  of  cost  data  available  from  the  NWS 
accounting  system,  we  did  not  attempt  to  determine  the  cost  to 
maintain  and  operate  the  system.  Our  review  focused  on  the 
costs  to  develop  AFOS.  We  have  made  use  of  NWS  figures  for 
operations  and  maintenance,  making  clear  that  they  were  provided 
by  NWS.  The  only  adjustment  we  have  made  was  to  allocate  over¬ 
head  to  the  NWS  figures  as  required  by  NOAA  and  NWS  accounting 
procedures . 

NWS  stated  that  it  disagreed  with  our  position  that  over 
$100  million  scheduled  to  be  spent  in  small  pieces  over  8  years 
could  be  accumulated  for  the  procurement  of  a  new  system.  We 
do  not  assume  or  state  that  this  is  the  case.  A  new  system 
development  project  should  be  separately  approved  and  funded  by 
the  Congress  and  not  internally  reprogramed  from  operations  and 
maintenance  funds. 
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Our  continuing  concern  is  that  NWS  has  not  adequately  deter¬ 
mined  the  most  cost-effective  course  of  action.  Further,  NWS  has 
not  adequately  demonstrated  that  splintering  its  scarce  personnel 
resources  across  three  separate  projects — maintaining  and  develop¬ 
ing  AFOS,  maintaining  the  current  system,  and  developing  a  new 
system — is  cost  effective  and  feasible. 

NWS'  comments  are  summarized  in  appendix  I  and  presented  in 
their  entirety  in  appendix  II. 
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CHAPTER  3 


THE  MANAGEMENT  OF  AFOS  HAS  BEEN  INEFFECTIVE 

A  direct  cause  of  many  AFOS  technical  and  operational 
problems  described  in  this  report  has  been  the  absence  of  a  co¬ 
herent  approach  to  managing  this  large  and  complex  effort. 

Most  of  the  delays  and  increased  costs  associated  with  the  AFOS 
project  have  occurred  because  NWS  did  not  (1)  establish  clear 
responsibility  for  AFOS  in  a  single,  full-time  manager,  (2)  use 
standard  approaches  to  managing  complex  ADP  projects,  and  (3) 
recognize  the  need  to  seek  additional  outside  assistance  in 
developing  AFOS.  NWS  needs  to  address  these  management  deficien¬ 
cies  regardless  of  the  ADP  system  it  develops  and  installs. 

BETTER  PROJECT  MANAGEMENT  IS  NEEDED 

For  over  12  years  we  have  reported  on  the  problems  associ¬ 
ated  with  developing  software  application  systems  in  the  Federal 
Government.  About  $300  million  in  waste  was  identified  in  these 
development  efforts.  We  reported  that  this  waste  of  money  and 
effort  could  have  been  mitigated  through  adherence  to  generally 
accepted  management  principles  such  as  the  following: 

— Developing  comprehensive  project  plans  that  address  all 
aspects  of  the  system  and  tie  in  with  other  agency 
software  plans. 

— Assigning  project  managers  as  the  central  point  of 
authority  for  major  software  development  efforts. 

— Preparing  realistic  cost  estimates  and  economic  analyses. 

— Establishing  effective  procedures  to  compare  a  system’s 
progress  with  the  approved  cost,  schedule,  and  performance 
estimates . 

The  AFOS  project  has  not  adhered  to  these  principles.  NWS 
made  plans,  but  they  were  not  often  implemented  or  followed.  In 
developing  AFOS,  NWS  did  not  develop  comprehensive  plans  address¬ 
ing  all  aspects  of  the  system  and  its  effect  on  other  NWS  systems 
and  projects.  Also,  NWS  did  not  establish  an  effective  project 
management  office  or  use  economic  analysis  as  an  effective  tool 
for  managing  and  controlling  AFOS  development. 

AFOS  has  been  directed  and  developed  by  staff  members 
who  are  involved  only  part  time  and  have  not  been  relieved  of 
their  normal  responsibilities.  These  part-time  staff  members  are 
required  to  balance  their  primary  duties  against  responsibilities 
for  AFOS.  Acquiring  the  knowledge  needed  to  develop  a  system  as 
complex  as  AFOS,  and  then  carrying  out  its  development,  demands  a 
greater  time  commitment  than  AFOS  staff  members  are  able  to  pro¬ 
vide  under  present  circumstances.  This  problem  has  resulted  in  a 
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lack  of  accountability,  unclear  lines  of  authority,  and  a  staff 
that  has  at  times  worked  at  cross-purposes. 


Major  AFOS  decisions  are  left  to  the  Director  of  NWS  or  his 
deputy,  but  these  officials  are  occupied  by  wide-ranging  respon¬ 
sibilities  and  cannot  be  expected  to  manage  the  project  on  a  day- 
to-day  basis.  The  result  is  that  AFOS'  problems  have  not  been 
addressed  in  a  timely  or  effective  manner. 

NWS  is  aware  of  these  problems.  In  a  March  19,  1981, 
memorandum,  the  AFOS  project  manager  stated: 

"Many  of  our  problems  can  be  traced  to  failures  of 
management  and  systems  discipline.  These  have  led 
to  unrealistic  plans,  schedules  and  expectations. 

Priorities  in  allocation  of  staff  and  resources  have 
been  obscure,  confusing  and  inconsistent  throughout 
the  organization.  The  flow  of  up-to-date,  authori¬ 
tative,  honest  information  has  been  spotty  and  slow, 
and  the  exercise  of  management  direction  often  has 
been  weak  and  fuzzy.  Collegial  decision-making  has 
promoted  participation  at  the  expense  of  focus, 
clarity  and  decisiveness." 

NWS  has  used  a  decentralized 
approach  to  develop  AFOS 

NWS  normally  operates  under  a  decentralized  management 
structure.  It  allows  its  field  offices  independence  in  carrying 
out  their  responsibilities,  and  no  NWS'  major  headquarters  or 
field  office  has  authority  over  any  of  the  others.  The  Office 
of  the  Deputy  Director  of  NWS  is  the  lowest  level  at  which  dis¬ 
putes  can  be  mediated  and  NWS-wide  policies  established  or 
enforced . 

AFOS  was  expected  to  produce  substantial  changes  in  NWS' 
day-to-day  operations.  For  this  reason,  a  well-intentioned  but 
ill-advised  attempt  was  made  to  share  responsibility  for  AFOS 
development  among  as  many  people  as  possible.  Thirteen  major 
NWS  offices  were  assigned  various  responsibilities  for  devel¬ 
oping  AFOS.  These  responsibilities  were  in  turn  often  shared 
with  several  other  offices.  For  example,  4  offices  were  re¬ 
sponsible  for  system  development  and  experimentation,  5  offices 
were  responsible  for  coordinating  AFOS  with  external  systems, 
and  12  offices  were  responsible  for  training.  In  addition, 
several  committees  were  established  at  NWS  headquarters  for 
program  coordination  and  problem  solving. 

The  Deputy  Director  of  NWS  was  initially  responsible  for 
directing  the  activities  of  all  people  and  offices  involved 
with  AFOS.  However,  high  turnover  in  this  position  led  NWS  to 
assign  the  responsibility  to  others  during  the  course  of  the 
project.  Currently,  project  management  is  assigned  to  the  di¬ 
rector  of  an  NWS  headquarters  office,  but  this  individual  is 


not  sufficiently  high  up  in  the  organization  to  enforce  policies 
agencywide,  and  most  of  the  staff  members  responsible  for  devel¬ 
opment  do  not  report  directly  to  him. 

AFOS  decentralization  has  created 
development  problems 

Because  of  NWS'  decentralized  management  approach  to  AFOS, 
key  managers  frequently  lack  the  authority  to  carry  out  their 
responsibilities  or  enforce  their  development  decisions.  An  ex¬ 
ample  of  unclear  lines  of  authority  is  NWS ‘  handling  of  the  AFOS 
air  conditioning.  Field  sites  experienced  systems  failures  which 
they  attributed  to  heat  problems.  When  NWS  engineers  investigated 
the  problem,  they  determined  that  field  sites  did  not  require  spe¬ 
cial  air  conditioning,  because  defective  internal  fans  were  caus¬ 
ing  the  problem,  and  directed  that  air  conditioning  not  be  pur¬ 
chased.  In  spite  of  this  decision,  NWS  regional  managers  proceeded 
to  install  air  conditioning  because  they  believed  it  was  necessary. 
In  short,  a  directive  issued  by  the  office  having  responsibility 
for  engineering  decisions  had  no  effect  on  the  regions'  purchasing 
decisions.  In  addition,  no  NWS  official  short  of  the  Director  has 
sufficient  authority  to  enforce  AFOS  development  directives. 

Our  primary  concern  is  that  offices  with  responsibility  for 
AFOS  decisions  have  no  authority  to  enforce  their  policies.  This 
lack  of  coordination  has  also  prevented  NWS  from  achieving  poten¬ 
tial  savings  through  a  consolidated  single-purchase  contract.  For 
example,  management  had  no  assurance  that  offices  purchased  suffi¬ 
cient  and  appropriate  air  conditioning  to  meet  their  needs. 

Without  a  strong  central  management  office  with  sufficient 
authority,  the  NWS  project  structure  precluded  effective  coordina¬ 
tion  and  accountability  in  systems  development.  Because  of  these 
problems : 

— Top  management  received  inadequate  and  misleading  informa¬ 
tion  on  the  time  and  resources  required  to  complete 
development  and  on  the  quality  of  the  system. 

— Development  priorities  were  not  clearly  established  and 
top  priority  work  was  ignored  in  favor  of  lower  priority 
work.  For  example,  monitoring  software  provides  necessary 
controls  over  the  status  of  all  field  office  systems  on 
each  regional  loop  at  all  times.  This  information  is 
critical  to  taking  corrective  action  when  a  field  office 
system  malfunctions.  Nevertheless,  NWS  programers  were 
removed  from  developing  the  SMCC's  monitoring  software,  a 
critical  AFOS  requirement,  and  assigned  to  the  development 
of  enhancements  which  are  valuable  but  not  critical  to  the 
system's  operation. 

— Programers  acted  independently  of  the  system  managers 
and  developed  software  not  approved  for  the  system. 

For  example,  numerous  enhancements  requested  by  field 
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personnel  were  developed  and  incorporated  into  the 
system  without  being  reviewed  or  approved  by  project 
management . 

— Resources  were  frequently  wasted  because  decisions  were 
based  on  unrealistic  completion  dates.  For  example, 
the  director  of  the  western  region  informed  the  Director 
of  NWS  in  September  1980  that  the  western  region  had 
expended  substantial  resources  preparing  for  a  precom¬ 
missioning  test  based  on  assurances  that  the  development 
staff  at  headquarters  could  meet  promised  completion 
dates.  The  test  slipped  because  the  headquarters  staff 
missed  its  completion  dates  and  failed  to  provide  the 
promised  support. 


AFOS  development  personnel 
work  at  cross-purposes 


AFOS  development  personnel  assigned  to  headquarters  and 
field  offices  report  to  the  heads  of  their  respective  offices 
rather  than  to  a  central  AFOS  project  manager.  This  reporting 
structure  has  resulted  in  work  that  was  done  at  cross-purposes. 
For  example,  the  two  regions  most  actively  involved  in  develop¬ 
ing  AFOS  assigned  a  top  priority  to  making  it  operational.  Yet, 
during  the  past  3  years  headquarters  has  stressed  the  importance 
of  testing  and  validating  the  AFOS  system  prior  to  going  opera¬ 
tional.  The  two  approaches  are  contradictory:  one  attempts  to 
determine  status  and  problems;  the  other  attempts  to  go  opera¬ 
tional  as  quickly  as  possible  and  analyze  the  system  later. 


This  situation  placed  a  significant  burden  on  headquarters 
development  staff,  who  had  planned  to  test  and  document  AFOS  be¬ 
fore  developing  it  any  further.  These  individuals  were  pressured 
by  field  personnel  to  develop  new  software  and  procedures  required 
for  operations. 


Another  example  of  poor  coordination  occurred  in  developing 
key  software.  The  western  region  believed  a  key  cause  of  AFOS' 
problems  was  the  inability  of  the  System  Monitoring  and  Coordina¬ 
tion  Center  to  monitor  the  status  of  the  telecommunications 
network — that  is,  to  know  which  field  offices  are  operational 
and  which  have  failed  and  been  cut  out  of  the  network.  Head¬ 
quarters  development  staff  assigned  this  function  a  lower  prior¬ 
ity.  As  a  result,  the  western  region  expended  considerable  re¬ 
sources  developing  software  designed  to  accomplish  this  task 
and  intended  for  use  at  the  SMCC.  When  the  SMCC  attempted  to 
use  the  software,  it  did  not  work.  The  software  will  require 
a  complete  redevelopment  to  make  it  operational. 


The  lack  of  success  with  this  specialized  software  is  not 
surprising.  The  programer  developing  it  in  the  western  region 
did  not  have  access  to  SMCC  personnel  who  were  familiar  with  the 
equipment  on  which  the  software  would  operate.  The  programer 
lacked  the  required  knowledge  and  experience  of  the  SMCC’s  unique 


22 


monitoring  and  control  operation  and  did  not  have  the  proper  test¬ 
bed  for  developing  the  software.  The  same  resources  expended  by 
a  programer  working  at  the  SMCC  rather  than  in  a  field  office, 
and  furnished  with  the  required  information  and  test  facilities, 
would  probably  have  produced  a  more  useful  product. 

NWS  DID  NOT  APPLY  ACCEPTED  APPROACHES 
TO  MANAGING  LARGE  PROJECTS 

The  National  Bureau  of  Standards  has  developed  and  issued 
guidance  for  Federal  agencies  in  developing  ADP  systems.  These 
standards  are  similar  to  those  used  by  the  private  sector  in 
systems  development.  They  are  widely  accepted. 

Although  NWS  recognized  the  need  to  use  standard  project 
development  techniques,  it  did  not  properly  utilize  these  tech¬ 
niques  in  developing  AFOS.  For  example,  management  was  not  pro¬ 
vided  with  timely  and  useful  project  status  reports  and  adequate 
cost  information  to  aid  in  decisionmaking,  and  most  of  the  cost 
information  available  was  in  error. 

Similarly,  the  use  of  management  reviews  was  of  limited 
value  because  insufficient  information  was  available  to  the 
project  manager  and  the  Director  of  NWS.  For  example,  headquar¬ 
ters  management  recently  reviewed  the  need  to  dedicate  one  person 
at  each  field  office  to  work  full  time  on  AFOS  maintenance  and 
operation.  However,  the  regions  had  already  assigned  a  full¬ 
time  staff  member  at  each  field  office,  in  some  cases  as  early 
as  2  years  before  the  headquarters  evaluation.  We  believe  this 
information  on  staff  assignments  should  have  been  available  to 
headquarters  management  since  the  action  of  regional  management, 
in  effect,  dedicated  over  100  staff  members  to  the  project.  Head¬ 
quarters  was  aware  of  the  heavy  commitment  of  field  resources 
but  not  the  extent  of  that  commitment.  The  absence  of  accurate 
information  on  actual  conditions  severely  reduces  the  value  of 
central  planning  and  management,  the  quality  of  the  decisions 
made,  and  management  efficiency  in  allocating  resources. 

AFOS  is  among  the  largest  nonmilitary  distributed  computer 
systems  ever  designed.  Developing  such  a  system  requires  a 
highly  trained  and  experienced  staff  of  managers,  designers,  and 
programers.  In  addition,  developers  should  have  the  necessary 
hardware  facilities  to  test  the  system.  However,  due  to  inexper¬ 
ience,  NWS  did  not  increase  its  staffing  to  accommodate  the  project. 
Many  of  the  NWS  headquarters  and  field  people  working  on  AFOS  have 
other  duties  and  are  providing  AFOS  support  only  on  a  part-time 
basis.  Further,  NWS  has  shifted  staff  members  from  one  ongoing 
task  to  another,  thereby  reducing  the  quality  of  their  work  and 
introducing  confusion  over  responsibility  and  accountability. 

NWS  had  not  previously  undertaken  any  ADP  development  proj¬ 
ects  of  the  size  and  scope  of  AFOS.  Further,  the  individuals 
who  have  served  as  project  manager  and  in  other  key  management 
positions  have  had  little  or  no  experience  in  managing  the 
development  of  large  ADP  systems. 


The  lack  of  experience  with  large  ADP  projects  was  a  sig¬ 
nificant  cause  of  NWS'  project  management  problems.  NWS  believed 
the  key  difficulty  in  developing  AFOS  was  the  system's  hardware. 

In  fact,  with  most  systems  the  crucial  difficulty  is  software 
development,  and  AFOS  is  no  exception.  Because  NWS  had  experience 
with  projects  like  radar,  it  concentrated  on  developing  hardware 
rather  than  software.  Further,  NWS'  belief  that  the  major  AFOS 
development  effort  would  be  hardware  contributed  to  its  decision 
to  manage  and  develop  the  system  in-house.  As  a  result,  AFOS 
has  been  managed  and  developed  by  people  who  are  not  sufficiently 
experienced  in  ADP  software  development. 

NWS  seriously  underestimated  the  size  of  the  AFOS  develop¬ 
ment  effort  and  as  a  result  allocated  inadequate  resources  to 
the  project.  Underestimating  time  and  resources  resulted  in  the 
following  problems: 

— NWS  installed  some  telecommunications  lines  earlier  than 
needed.  These  lines  had  to  be  removed  very  shortly  after 
installation  to  reduce  costs.  They  were  reinstalled  later. 

— The  AFOS  managers  made  several  decisions  based  on  arbitrary 
deadlines,  not  needs.  One  of  these  decisions,  not  design¬ 
ing  and  developing  the  SMCC  subsystem  to  meet  its  special¬ 
ized  monitoring  and  control  needs,  continues  to  prevent 
successful  software  development  today. 

— NWS  provided  unrealistic  completion  estimates  to  the  field 
resulting  in  increased  costs,  morale  problems,  and  a  loss 
of  credibility.  For  example,  NWS'  plans  allocated  insuffi¬ 
cient  time  for  developing,  testing,  and  installating  the 
software  and  imposed  unrealistic  deadlines.  Actual  devel¬ 
opment  time  was  about  4  years.  Because  of  these  unrealistic 
deadlines,  developers  disregarded  testing,  documenting,  and 
other  development  procedures  in  an  effort  to  meet  deadlines. 

— NWS  assigned  responsibilities  to  developers  who  lacked 
the  training  and  experience  to  carry  them  out.  For  ex¬ 
ample,  the  programers  who  modified  the  hardware  operating 
system,  probably  one  of  the  most  complex  of  software 
efforts,  had  limited  experience  with  writing  operating 
systems . 

INEFFECTIVE  USE  OF  CONTRACTORS 

NWS  has  not  made  effective  use  of  consultants  and  associated 
studies  that  it  commissioned  in  developing  AFOS  and  has  not  made 
appropriate  changes  in  its  management  structure  when  consultants 
advised  it  to  do  so.  Had  certain  consultant  recommendations  been 
adopted,  AFOS'  problems  could  have  been  mitigated.  For  example, 
in  1979  NWS  engaged  one  contractor  to  assess  the  AFOS  software 
design  and  another  to  review  the  remainder  of  the  AFOS  program 
in  greater  depth.  Its  recommendations  included: 
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— Stopping  the  entire  AFOS  development  and  establishing 
a  formal  management  structure  before  proceeding. 

— Developing  only  a  limited  test  network  and,  in  parallel, 
redoing  the  AFOS  system  from  the  top  down  before 
going  operational. 

— Reordering  the  priority  of  system  development  activities. 

In  our  opinion,  it  is  difficult  for  managers  and  staff  who 
are  primarily  trained  in  meteorology  to  develop  a  complex,  state- 
of-the-art,  distributed  minicomputer  system.  We  have  discussed 
these  concerns  with  the  Director  of  NWS  and  other  agency  offi¬ 
cials.  It  is  our  position  that  contractors  experienced  in 
developing  major  ADP  systems  should  be  engaged  for  the  new  de¬ 
velopment  effort,  and  NWS  should  contract  out  development  work 
beyond  its  internal  capabilities  and  concentrate  NWS  resources 
on  requirements  analyses,  system  definitions,  and  managing  the 
contractor's  performance. 

We  believe  that  greater  reliance  on  outside  contractors 
could  substantially  improve  NWS'  success  in  developing  a  new 
system.  This  should  also  relieve  the  staffing  problem  because 
fewer  NWS  people  would  be  involved  as  the  contractor  performs 
most  of  the  time-consuming  technical  work.  Further,  contract¬ 
ing  should  improve  the  quality  of  the  system  because  the  actual 
developers  will  be  personnel  trained  and  experienced  in  ADP. 

The  main  thrust  of  recommendations  from  several  studies  was 
that  NWS  should  establish  a  strong  project  management  office, 
increase  systems  discipline,  use  standard  project  and  software 
development  procedures,  and  completely  redevelop  large  portions 
of  AFOS.  The  agency,  however,  finds  it  unacceptable  to  impose 
a  strong  project  management  office  on  NWS'  decentralized  manage¬ 
ment  structure  and  has  not  adopted  the  consultants'  recommenda¬ 
tions  on  the  need  for  an  improved  structure. 

CONCLUSIONS 


NWS  did  not  establish  an  effective  project  management  office 
with  a  full-time  project  manager  having  complete  authority  for 
development.  Project  personnel  were  assigned  part  time  and  some¬ 
times  worked  at  cross-purposes.  NWS  did  not  adequately  assess 
the  magnitude  and  complexity  of  the  AFOS  project  and  therefore 
did  not  assign  a  full-time  manager  with  authority  equal  to  the 
responsibilities  of  the  assignment. 

The  result  of  this  lack  of  clearly  defined  responsibility 
for  AFOS  has  been  NWS'  inability  to  effectively  manage  the  project. 
NWS  is  a  highly  decentralized  organization  which  permits  wide 
latitude  to  headquarters  and  field  staffs.  Although  this  organi¬ 
zational  arrangement  may  be  acceptable  for  other  NWS  activities, 
it  is  incompatible  with  developing  ADP  systems  such  as  AFOS.  In 
developing  a  new  system,  NWS  needs  to  address  this  problem  if  the 
development  is  to  be  successful. 
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NWS  neglected  to  apply  basic  systems  management  principles 
to  AFOS.  NWS'  use  of  project  management  techniques  wa*  inef¬ 
fective  because  managers  lacked  experience,  adequate  >.  orce- 
ment,  and  feedback  systems. 

RECOMMENDATIONS 

We  recommend  that  the  Secretary  of  Commerce  direct  NWS  to 

— establish  a  project  management  office  and  assign  all 
development  personnel  to  that  office  on  a  full-time 
basis  in  completing  AFOS  development  and  in  developing 
a  new  system; 

— appoint  a  project  manager  with  clear  authority  for 
AFOS  and  for  the  planned  new  system; 

— select  and  enforce  standard  software  development 

procedures,  including  documentation  and  testing  for  the 
new  system;  and 

— contract  out  system  development  activities  which  exceed 
in-house  development  capabilities. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

NWS  stated  that  it  has  already  implemented  the  recommenda¬ 
tions  in  this  chapter  that  apply  to  the  current  AFOS  project  and 
that  its  future  plans  related  to  managing  the  development  of  a 
replacement  for  AFOS  reflect  our  recommendations. 

NWS  stated  that  it  has  established  a  project  management 
office  for  AFOS  and  that  it  has  assigned  all  development  person¬ 
nel  to  that  office  on  a  full-time  basis.  Further,  it  stated  that 
it  has  appointed  a  project  manager  with  clear  authority  for  the 
AFOS  project.  In  developing  the  new  system,  NWS  plans  to  adopt 
and  enforce  standard  software  development  procedures.  These 
procedures  would  include  documentation  and  testing  of  the  new 
system.  For  the  new  system  development  project,  NWS  agreed  it 
should  engage  a  contractor  with  experience  in  complex  and  com¬ 
prehensive  computer/telecommunications  systems  development 
projects. 

In  our  view,  the  scope  and  magnitude  of  NWS'  effort  to 
automate  requires  even  greater  emphasis  on  strong  centralized 
management  than  shown  by  NWS  to  date.  Specifically,  the  manage¬ 
ment  of  a  project  like  AFOS  should  include  individuals  with 
experience  in  managing  large  and  complex  ADP  telecommunications 
projects.  Having  the  project  manager  report  to  the  Director  of 
NWS  does  not  in  itself  adequately  address  the  problem  of  manag¬ 
ing  and  controlling  a  complex  and  comprehensive  project  like 
AFOS.  Our  review  disclosed  that  the  AFOS  project  manager  had 
only  limited  control  over  the  resources  critical  to  project 
development.  Specifically,  personnel  reported  to  their  normal 
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offices;  were  evaluated  by  managers  other  than  the  AFOS  project 
manager;  and,  to  a  large  extent,  were  assigned  to  AFOS  on  a 
part-time  basis. 

NWS  also  stated  that  AFOS  project  personnel  continue  to 
report  to  and  be  rated  by  their  normal  supervisors  and  not  the 
AFOS  project  manage*' .  This  is  a  primary  problem  in  NWS*  man¬ 
agement  of  AFOS.  Ir  these  personnel  are  in  fact  assigned  to 
AFOS  development  on  a  full-time  basis,  they  should  report  to 
the  AFOS  project  manager  and  not  to  their  normal  supervisors. 

We  agree  with  NWS’  statement  that  management  deficiencies 
are  not  valid  reasons  for  not  using  a  system  after  technical 
problems  have  been  resolved.  Our  report  concludes  that  NWS 
should  defer  implementing  AFOS  on  the  basis  of  technical  defi¬ 
ciencies  and  not  because  of  management  problems.  However,  our 
report  also  points  out  the  extent  to  which  management  deficien¬ 
cies  contributed  to  the  technical  deficiencies. 

NWS’  statements  regarding  its  reorganization  of  the  AFOS 
management  structure  do  not  adequately  address  our  concern  for 
lack  of  centralized  control  over  AFOS  development.  The  reorgani¬ 
zation  of  AFOS  management  continues  NWS'  policy  of  the  project 
manager  functioning  as  a  coordinator  of  work  performed  in  a  num¬ 
ber  of  different  offices.  It  still  does  not  provide  a  single 
source  for  decisionmaking  on  AFOS  and  it  does  not  assign  full-time 
development  staff  to  a  project  manager  who  can  directly  manage 
their  activities.  Therefore,  the  need  for  a  strong  project  man¬ 
agement  office  with  complete  authority  to  make  key  decisions 
remains  unfulfilled. 

The  extent  and  effect  of  AFOS  management  deficiencies  has 
been  assessed  several  times  during  the  life  of  the  project.  In 
1979  NWS  contracted  with  two  ADP  consulting  firms  for  comprehen¬ 
sive  reviews  of  AFOS.  Both  reviews  were  completed  in  late  1979. 
The  first  report's  assessment  of  NWS  management  and  its  structure 
was  as  follows. 

"The  AFOS  program  is  of  sufficient  magnitude  to  warrant 
a  distinct  project  management  office  with  sufficient 
authority  to  obtain  binding  commitments  from  support 
organizations  within  the  NWS  structure.  Because  of 
these  organizational  deficiencies,  the  day-to-day 
business  of  AFOS  program  management  became  diffused 
throughout  the  NWS,  and  program  commitments  became 
entangled  with  other  NWS  day-to-day  business.  There 
was  no  driving  force — a  project  management  office — to 
maintain  program  momentum,  to  provide  authoritative 
leadership,  and  to  be  accountable  for  the  job." 

The  second  review  by  a  contracting  firm  summarized  its 
concerns  with  AFOS  management  as  follows. 
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"The  most  critical  need  is  the  establishment  of  a 

strong  and  permanent  management  structure  for  AFOS." 

Our  review,  initiated  about  1  year  after  these  reports, 
identified  similar  management  and  organizational  deficiencies. 

In  short,  the  AFOS  project  management  structure  provides  little 
overall  control  and  guidance  to  the  system  development  activi¬ 
ties.  These  conditions  continue  to  exist,  and  their  effect  ham¬ 
pers  progress  toward  developing  a  new  system  to  meet  agency 
needs . 

In  this  report  we  state  the  need  for  a  contractor’s  exper¬ 
tise  to  develop  complex  technical  systems  such  as  AFOS.  We  still 
contend  that  the  technical  work  of  developing  major  ADP  projects 
should  be  accomplished  with  personnel  who  have  ADP/telecommunica- 
tions  training  and  experience  in  developing  complex  systems,  as 
opposed  to  using  meteorologists  who  lack  this  experience. 

NWS  noted  in  its  response  that  it  did  use  contractors  for 
AFOS  development.  NWS  did  hire  a  number  of  contract  programers 
who  were  used  in  AFOS  development  under  the  direction  of  NWS 
managers.  They  were  of  considerable  assistance  to  NWS.  However, 
given  the  size  of  AFOS  and  the  limited  amount  of  outside  assist¬ 
ance  used,  it  did  not  adequately  address  NWS'  staffing  problems. 
Major  ADP  development  projects  simply  require  too  much  staff  time 
to  be  developed  without  extensive  contractor  assistance  or  hiring 
of  full-time  ADP  development  personnel. 

NWS  also  said  that  management  must  retain  the  responsibility 
of  NWS.  We  completely  agree.  However,  using  contractor  personnel 
to  perform  technical  tasks  does  not  diminish  NWS'  management 
authority. 

We  note  that  NWS  has  taken  steps  to  implement  two  of  the 
recommendations  in  this  chapter  and  plans  to  implement  the  remain¬ 
ing  two  when  it  develops  a  new  system.  We  continue,  however,  to 
have  reservations  about  the  adequacy  of  NWS'  changes  to  its  man¬ 
agement  structure.  There  remains  a  crucial  need,  still  not  recog¬ 
nized  by  NWS,  to  develop  a  stronger  central  AFOS  project  manage¬ 
ment  structure. 

NWS  stated  that  different  personnel  are  involved  in  develop¬ 
ing  a  new  system  and  in  operating  AFOS.  As  a  result,  NWS  be¬ 
lieves  it  can  concurrently  develop  a  new  system,  operate  AFOS, 
and  maintain  the  current  system.  NWS  has  not  fully  acknowledged 
that  during  AFOS'  very  lengthy  and  costly  development  phase, 
field  operating  personnel  invested  vast  amounts  of  time  in  de¬ 
veloping,  tejsting,  and  operating  the  system.  This  time  expendi¬ 
ture  is  likply  to  be  repeated  when  NWS  is  again  involved  in  » 
new  system  ^development  effort.  Managing  and  developing  AFOS  has 
already  strained  NWS  field  and  headquarters  personnel.  Devel¬ 
oping  a  new  system,  in  addition  to  supporting  AFOS  and  maintaining 
the  current  system,  will  place  an  unrealistic  burden  on  both  man¬ 
agement  and  technical  personnel.  Further,  by  not  adequately 
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assessing  the  total  personnel  needs  based  on  a  system  to  properly 
account  for  personnel  costs,  NWS  will  continue  to  ignore  the  very 
real  and  heavy  investment  of  its  personnel  in  AFOS  development. 

As  we  stated  in  chapter  2,  NWS  should  follow  the  management 
and  cost  accounting  procedures  outlined  in  OMB  Circular  A-109. 

This  minor  change  in  itself  would  help  management  recognize  the 
total  cost  of  development  and  provide  a  sounder  basis  to  deter¬ 
mine  personnel  costs  for  all  development  and  operating  activities. 
Over  the  next  several  years,  significantly  more  staff  resources 
will  be  required  to  operate  and  maintain  AFOS  than  those  needed 
to  operate  the  current  communications  system. 

NWS'  comments  are  summarized  in  appendix  I  and  presented  in 
their  entirety  in  appendix  II. 


CHAPTER  4 


THE  AFOS  SOFTWARE  PROBLEMS  PERSIST 


NWS'  problems  with  the  AFOS  software  have  been  a  major  cause 
of  delays.  The  software  to  which  NWS  is  committed  has  not  met 
all  the  requirements  initially  established  for  AFOS.  NWS'  cur¬ 
rent  and  final  attempt  to  remedy  some  of  the  software  deficien¬ 
cies  cannot  overcome  its  fundamental  defects.  NWS  did  not  take 
advantage  of  generally  accepted  software  development  procedures, 
including  documentation  and  basic  testing,  to  validate  the  system's 
design  and  performance. 

AFOS  SOFTWARE  CANNOT  MEET  NWS'  NEEDS 

Software  is  defined  as  a  detailed  set  of  instructions  which 
controls  hardware  and  enables  the  computer  system  to  manipulate 
information  and  carry  out  user  tasks.  As  hardware  costs  con¬ 
tinue  to  decrease,  software  has  become  the  most  costly  item  in 
ADP  systems.  The  AFOS  software  will  require  continual  mainte¬ 
nance  and  modification  to  meet  changing  user  needs  and  new 
requirements . 

In  the  AFOS  system,  software  is  divided  into  three  categor¬ 
ies — applications,  operating  system,  and  telecommunications.  The 
application  software  includes  a  set  of  instructions  that  produces 
products  or  outputs  for  end  users.  The  AFOS  operating  system 
software  is  housed  in  the  minicomputers  to  control  and  direct 
the  computer  to  carry  out  various  functions.  The  third  type  of 
AFOS  software  controls  and  directs  the  functions  of  the  telecom¬ 
munications  system  to  transmit  data  over  the  telephone  lines  to 
and  from  a  weather  station. 

According  to  the  AFOS  functional  specifications,  the  software 
was  to  meet  three  major  requirements:  (1)  take  information  from 
the  telecommunications  system  and  store  it  for  developing  fore¬ 
casts,  (2)  retrieve  data  stored  in  the  system  so  that  forecasters 
can  analyze  it,  and  (3)  minimize  the  work  involved  in  composing 
and  sending  weather  information  to  users.  In  addition  to  meeting 
these  broad  requirements,  AFOS  was  planned  to  include  other 
characteristics.  The  system  was  to 

— be  highly  reliable  (NWS  initially  specified  that  the  soft¬ 
ware  should  not  malfunction  more  than  once  a  year  (see 
footnote  on  p.  39)), 

— be  flexible  enough  to  accommodate  future  system  enhance¬ 
ments  , 

— be  able  to  provide  backup  capability  in  the  event  of  a 
malfunction , 
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— support  multiple  users  of  the  system  at  each  site  on  a 
concurrent  basis, 

— provide  quick  response  time  to  users,  and 

— be  simple  enough  in  operation  that  it  could  be  maintained 
by  forecasters  with  minimal  ADP  experience  and  training. 

The  software  developed  for  AFOS  does  not  meet  these  charac¬ 
teristics  initially  specified  by  NWS.  Further,  the  software 
lacks  the  capabilities  to  meet  minimal  system  requirements, 
frequently  fails,  requires  extensive  staff  time  to  support,  and 
requires  a  trained  computer  specialist  onsite  to  maintain  opera¬ 
tions  at  an  acceptable  level. 

We  believe  a  primary  cause  of  NWS'  software  problems  is  its 
inexperience  with  large  software  development  projects.  In  part, 
this  problem  is  also  caused  by  the  changes  occurring  in  ADP 
technology.  In  the  early  days  of  computers,  the  price  of  the 
equipment  (hardware)  was  the  major  ADP  cost.  Also,  the  computer 
programs  (software),  which  make  the  equipment  operate,  were  re¬ 
latively  inexpensive.  However,  software  now  costs  considerably 
more  than  hardware,  which  has  steadily  declined  in  price  because 
of  technological  advances.  We  noted  in  our  recent  report,  "Wider 
Use  of  Better  Computer  Software  Technology  Can  Improve  Management 
and  Reduce  Costs"  ( FGMSD-80-38 ,  Apr.  29,  1980),  that  recent  studies 
predict  that  by  1985,  over  90  percent  of  the  cost  of  ADP  will  be 
attributable  to  software.  Our  discussions  with  NWS  managers  and 
developers  made  clear  that  NWS  believed  the  higher  priority  task 
in  developing  AFOS  was  the  design  and  procurement  of  its  hardware. 
As  a  result,  procedures  for  software  development  and  their  enforce¬ 
ment  were  accorded  a  lower  priority. 

The  software  is  unnecessarily  complex 

The  AFOS  software  is  complex  and  tightly  integrated  with 
the  telecommunications  and  hardware  operating  system.  Of  neces¬ 
sity  the  software  must  have  some  complexity,  but  due  to  hardware 
and  operating  system  limitations,  the  software  is  much  more  com¬ 
plex  than  it  needs  to  be. 

Software  is  normally  developed  in  independent  modules  which 
are  tied  to  other  modules  with  a  limited  number  of  interconnec¬ 
tions.  The  point  of  interconnection  between  two  modules  is 
called  an  execution  path;  it  designates  the  order  in  which  the 
software  performs  a  set  of  instructions.  Modules  are  designed  to 
be  independent  with  very  few  interconnections  so  that  changes  in 
one  module  do  not  cause  changes  in  other  modules.  Also,  the  use 
of  independent  modules  greatly  increases  the  ease  of  isolating 
problems  and  program  errors. 

However,  to  resolve  operating  system  and  capacity  problems, 
AFOS  programers  used  a  highly  integrated  software  design  to  con¬ 
serve  core  memory  and  designed  software  modules  to  perform  mul¬ 
tiple  functions.  They  also  closely  linked  the  major  software 
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subsystems  (that  is,  communications,  data  storage  and  retrieval, 
and  message  composition).  Thus,  the  AFOS  software  is  composed 
of  subsystems  made  up  of  highly  interdependent  software  modules. 
Consequently,  the  NWS  programing  staff  has  experienced  problems 
when  modifying  one  module  or  subsystem.  Because  of  these  flaws 
in  the  software  design,  it  is  most  difficult  to  avoid  problems 
with  other  parts  of  the  system. 

Because  the  original  operating  system  did  not  meet  AFOS 
requirements,  the  NWS  programing  staff  significantly  modified 
the  vendor's  operating  system  to  make  it  more  compliant.  How¬ 
ever,  these  modifications  have  also  greatly  increased  the  sys¬ 
tem's  complexity,  which  contributed  to  reliability  problems. 

AFOS  consists  of  four  major  subsystems  made  up  of  over  300 
closely  linked  software  modules.  The  NWS  programing  staff  esti¬ 
mates  that  when  the  software  system  is  operating,  it  has  millions 
of  potential  execution  paths.  With  this  number  of  modules  and 
potential  execution  paths,  tracking  software  routines  and  tasks 
to  resolve  software  problems  is  difficult  and  costly. 

The  complexity  of  the  software  has  created  numerous  unsolved 
deficiencies  that  affect  all  major  subsystems.  Foremost  among 
these  deficiencies  is  the  deadlock  problem.  The  system  ex¬ 
periences  deadlock  when  the  computer  attempts  to  process  two 
or  more  tasks  that  need  the  same  resources.  This  condition 
makes  the  computer  incapable  of  processing  other  tasks,  and 
therefore  the  system  malfunctions.  Because  the  computer  is 
processing  data  from  many  sources  simultaneously,  the  problem 
occurs  frequently  and  unpredictably . 

Other  software  problems  include  the  following: 

— When  more  than  one  forecaster  at  a  site  uses  the  system, 
the  data  retrieval  and  manipulation  routines  fail. 

— Of  the  four  major  software  backup  routines,  only  one 
works  reliably,  but  with  extensive  use  it  causes  the 
data  storage  subsystem  to  malfunction. 

— When  the  message  composition  subsystem  fails  while  a 
forecaster  is  preparing  a  forecast,  the  prepared  mes¬ 
sage  is  frequently  lost.  The  forecaster  must  then 
rewrite  the  forecast. 

NWS'  attempts  to  correct  software  problems 

NWS  has  adopted  numerous  approaches  in  trying  to  solve  the 
software  deficiencies,  but  they  are  so  integrated  with  the  hard¬ 
ware  capacity  and  operating  system  problems  that  solutions  applied 
to  date  have  been  inadequate.  In  February  1981,  following  a  2- 
month  assessment  of  the  system,  NWS  concluded  that: 

— The  AFOS  software  is  unable  to  meet  NWS'  operational 
requirements . 
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— The  software  as  developed  does  not  provide  an  adequate 
basis  for  continuing  development. 

— Any  further  software  development  would  not  be  cost 
effective  and  probably  would  not  result  in  significant 
improvements . 

NWS  DID  NOT  ADEQUATELY  USE  STANDARD 
SOFTWARE  DEVELOPMENT  PROCEDURES 

NWS  did  not  follow  standard  software  development  procedures 
established  by  the  Nation:!  Bureau  of  Standards  for  Federal 
agencies  to  ensure  the  development  of  quality,  documented  sys¬ 
tems  that  meet  user  needs  and  are  efficient  and  cost  effective. 
For  example,  NWS  did  not  (1)  adequately  identify  and  freeze  the 
system's  functional  requirements,  (2)  prepare  design  specifica¬ 
tions,  (3)  implement  a  change  control  process,  (4)  develop  the 
system  as  independent  modules,  (5)  document  the  system,  and  (6) 
develop  a  comprehensive  testing  program. 

As  a  result  of  not  adhering  to  these  procedures,  NWS  pro- 
gramers  developed  the  software  system  without  a  clear  understand¬ 
ing  of  its  performance  requirements.  The  resulting  system  is 
poorly  understood  by  the  programers  responsible  for  its  develop¬ 
ment  and  is  for  all  practical  purposes  completely  undocumented. 

In  addition,  the  subsystems  are  so  interdependent  that  attempts 
to  improve  the  software  often  result  in  unanticipated  problems 
elsewhere  in  the  system.  To  resolve  these  new  problems,  further 
system  refinements  are  required  that  in  turn  may  introduce  new 
software  problems.  As  a  result,  the  development  has  been  time 
consuming  and  costly  and  has  produced  software  that  does  not  work 
These  problems  could  have  been  avoided  or  mitigated  by  rigorously 
utilizing  standard  software  development  procedures. 

Requirements  were  not  defined  and  frozen 


Functional  requirements  identify  the  work  a  software  system 
is  intended  to  perform  and  detail  specific  interrelationships 
among  the  system's  separate  parts.  When  these  requirements  have 
not  been  properly  defined,  the  software  designers  are  hampered  in 
developing  software  because  they  do  not  have  an  adequate  basis 
to  compare  their  results  against  needs.  The  developing  organiza¬ 
tion  will  experience  difficulties  in  managing  the  development 
process  because  the  resources  and  time  needed  to  accomplish  the 
total  task  cannot  be  determined.  Further,  it  becomes  difficult 
to  establish  milestones  because  there  is  no  assurance  that  devel¬ 
opment  estimates  are  reasonable. 

NWS  has  neither  adequately  identified  nor  frozen  system 
requirements.  The  system  requirements  have  been  significantly 
changed  since  the  project  was  initiated  in  1974.  From  1974  to 
1978,  users  identified  new  requirements  and  enhanced  capabilities 
that  were  subsequently  incorporated  into  the  system.  Since  1978, 
the  developers  have  been  reducing  system  capability  in  an  effort 
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to  achieve  a  stable  system.  In  effect,  NWS  is  now  removing 
much  of  the  software  it  has  spent  the  last  few  years  developing. 

As  a  result  of  NWS'  changing  requirements,  two  problems 
have  occurred. 

— Completed  development  work  had  to  be  redone  because 
of  design  changes. 

— The  system  designers  have  been  unable  to  effectively 
develop  the  system  because  it  is  continually  changing. 

This  situation  occurred  primarily  because  NWS  initially  believed 
that  the  system  had  more  than  adequate  capacity.  As  a  result, 
management  believed  it  could  respond  to  the  needs  of  the  field 
and  included  desired  enhancements. 


AFOS  lacked  adequate  design  specifications 
and  a  change  control  process 

A  primary  component  of  the  development  process  is  software 
design  specifications.  Design  specifications  translate  require¬ 
ments  into  the  detailed  guidance  needed  by  the  programers.  In 
addition,  these  specifications  provide  a  system  overview  that  per¬ 
mits  an  assessment  of  whether  the  design  is  feasible  or  desirable 
in  relation  to  the  total  system  objectives. 

Specifications  detail  what  each  part  of  the  system  will  do 
and  help  manage  the  development  in  three  ways.  First,  the  speci¬ 
fications  provide  a  system  overview  of  all  planned  capabilities 
for  management  analysis.  Second,  by  comparing  progress  against 
plans,  managers  can  monitor  development.  Third,  the  design 
specifications  provide  a  baseline  for  testing. 

After  developing  design  specifications,  the  organization 
should  enforce  a  change  control  process.  Basically,  this  process 
ensures  that  any  change  is  thoroughly  analyzed  with  respect  to  both 
specific  and  overall  system  requirements.  Further,  this  control 
process  should  limit  changes  to  the  absolute  minimum. 


NWS  developed  AFOS  without  the  benefit  of  either  software 
design  specifications  or  an  adequate  change  control  process. 
Consequently,  programers  prepared  computer  programs  directly  from 
functional  requirements  documents.  Information  from  these  docu¬ 
ments  is  inadequate  because  they  lack  the  specifics  required  to 
write  computer  programs. 


As  a  result  of  not  developing  specifications  and  implement¬ 
ing  a  change  control  process,  NWS  lost  control  of  the  development 
process.  Specifically: 


— Programers  determined  independently  what  programs  to 
develop  and  what  functions  the  programs  would  contain. 
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— AFOS  developers  do  not  have  sufficient  knowledge  of  what 
the  program  modules  contain  and  how  they  work. 

NWS  did  not  recognize  the  need  for  relatively  precise  design 
specifications.  NWS  understands  very  well  what  AFOS  is  to  do; 
this  understanding,  however,  is  limited  to  a  knowledge  of  the 
desired  end  result.  NWS  does  not  know  in  specific  terms  how 
the  system  is  to  perform  these  functions.  For  example,  NWS 
wants  all  information  available  to  field  offices  within  minutes 
of  the  information  entering  the  system.  This  general  knowledge, 
however,  in  terms  of  the  way  the  system  meets  the  need,  requires 
hundreds  or  thousands  of  detailed  interactions  between  software 
modules  in  different  subsystems.  Precise  design  specifications 
are  important  because  an  ADP  system's  software  will  usually 
undergo  almost  continuous  modification  and  change.  The  programer 
making  these  changes  must  clearly,  fully,  and  accurately  under¬ 
stand  the  system.  Without  design  specifications,  this  knowledge 
cannot  be  achieved. 

In  a  1981  software  technical  assessment,  NWS  concluded  that 
further  development,  no  matter  what  level  of  resources  are 
expended,  may  not  produce  significant  software  improvements.  1/ 
This  is  the  basis  of  NWS'  conclusion  that  the  software  must  be 
completely  rewritten.  The  lack  of  design  specifications  is  a 
key  reason  for  that  assessment. 

Lack  of  documentation 

Documentation  is  a  detailed  set  of  procedures  which  explains 
what  the  program  does,  how  it  performs  the  function,  how  the 
program  interacts  with  other  programs,  and  other  information. 

This  documentation  must  be  kept  current  to  make  it  possible  to 
modify  or  maintain  the  program  following  the  software  development 
phase.  Documentation  also  includes  instructions  and  procedures 
for  those  operating  the  system,  such  as  forecasters  at  field 
sites.  This  documentation  helps  in  training  personnel  to  operate 
AFOS  and  helps  the  operator  resume  normal  operations  following 
system  failures. 

Documentation  is  absolutely  critical  to  effective  software 
development  and  operation.  Without  adequate  documentation, 
development,  maintenance,  and  enhancement  become  difficult,  if 
not  impossible.  In  developing  AFOS,  NWS  did  not  develop  and 


1/The  NWS  assessment  that  further  software  development  may  not 
produce  improvements  is  based  primarily  on  the  lack  of  docu¬ 
mentation  and  modularity  in  the  original  design.  Because  of 
these  two  problems,  further  development  may  produce  unintended 
side  effects  which  make  the  software  worse,  not  better.  The 
only  way  to  resolve  these  two  problems  is  by  completely  rede¬ 
veloping  the  software  using  a  modular  design  and  documenting 
the  new  software.  This  is  in  essence  a  complete  software 
redevelopment  effort. 


enforce  documentation  standards  and  procedures.  As  a  result, 
documentation  was  not  prepared,  and  NWS  to  date  has  been  unable 
to  prepare  documentation  after  the  fact. 

In  1979  NWS  initiated  a  90-day  effort  to  document  the  system. 
NWS  program  officials  acknowledged,  and  we  concur,  that  this  effort 
produced  unsatisfactory  results  that  at  best  provide  a  high-level 
system  overview.  The  documentation  is  not  an  adequate  base  for 
system  reviews,  assessment,  or  development.  Because  of  the  size 
of  the  project  and  the  loss  of  key  personnel,  NWS,  in  our  opinion, 
cannot  adequately  document  the  system  on  a  catchup  basis.  As  a 
result,  NWS  cannot  completely  understand  the  contents  of  several 
hundred  thousand  lines  of  computer  program  code. 

This  deficiency  occurred  because  NWS  initially  believed  the 
software  development  effort  was  a  task  which  could  be  quickly 
accomplished.  This  belief  is  demonstrated  by  the  insufficient 
time  initially  scheduled  for  software  development.  Had  NWS 
recognized  the  magnitude  of  the  task  and  the  time  required  (4 
years),  we  believe  it  would  have  placed  greater  emphasis  on 
documentation . 

During  our  review  NWS  initiated  action  to  document  selected 
segments  of  AFOS'  software  by  obtaining  the  services  of  an  out¬ 
side  contractor.  As  of  October  1981,  this  material  was  not  avail¬ 
able  for  review.  Therefore,  we  are  not  in  a  position  to  comment 
on  the  adequacy  of  the  documentation  under  development.  However, 
it  has  generally  been  the  case  that  after-the-fact  documentation 
has  been  of  limited  value. 

Inadequate  software  testing 

Without  a  carefully  controlled  and  thorough  test  program, 
management  has  little  assurance  that  the  system  will  meet  user 
needs.  Further,  management  loses  a  critical  checkpoint  for 
identifying  and  correcting  system  problems  and  minimizing 
project  slippages.  An  adequate  test  program  requires 

— an  independent  testing  team  with  sufficient  authority  to 
develop  and  enforce  test  procedures, 

— an  adequate  staff  with  training  and  experience  in  ADP 
system  testing, 

— program  and  design  specifications  which  include  objectives 
and  measurable  performance  goals,  and 

— a  realistic  and  appropriate  test  methodology. 

The  NWS  testing  program  lacked  all  of  these  requirements. 

The  testing  team  had  limited  authority,  had  minimal  experience 
with  ADP  systems,  was  assigned  part  time  to  testing,  and  had  such 
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limited  resources  that  it  was  unable  to  effectively  carry  out 
its  tasks.  The  team  tested  known  problems  on  a  pass/fail  basis. 

If  the  program  would  execute,  it  was  good;  if  not,  ie  was  defi¬ 
cient.  The  testers  and  developers  lacked  program  specifications 
showing  the  functions  a  module  or  subsystem  should  perform  and 
how  it  fitted  into  the  total  system.  Without  this  performance 
criteria,  test  teams  can  only  use  an  empirical  methodology  (try 
it  and  see  what  it  does). 

Empirical  methodologies  are  not  considered  appropriate  for 
system  testing  because  they  do  not  provide  information  on  how  the 
program  operates  and  its  effect  on  other  programs.  As  a  result, 
this  approach  provides  inadequate  information  to  serve  as  a  base 
for  further  system  development.  The  approach  severely  reduced 
the  value  of  the  limited  tests  performed. 

The  NWS  test  manager  was  assigned  two  part-time  staff  members 
to  perform  all  AFOS  testing.  The  team  had  minimal  experience 
with  ADP  testing  and  lacked  the  training  required  for  this  func¬ 
tion.  Due  to  its  size  and  complexity,  AFOS  required  a  large, 
full-time  staff  for  an  adequate  test  program.  The  test  team 
should  have  had  experience  and  knowledge  in  designing  (1)  tele¬ 
communications,  (2)  software,  (3)  hardware,  (4)  operating  systems, 
and  (5)  software  testing  methodologies. 

NWS  SHOULD  CONTRACT  FOR 
SOFTWARE  DEVELOPMENT 

NWS  is  an  operational  organization  which  normally  is  not 
involved  in  developing  major  software  projects.  As  a  result, 

NWS  as  part  of  its  normal  business  does  not  require  personnel 
trained  and  experienced  in  the  various  disciplines  required  by 
software  projects.  In  addition,  as  the  Director  of  NWS  has  re¬ 
peatedly  pointed  out,  NWS  staff  resources  are  stretched  extremely 
thin  even  for  meeting  normal  organizational  requirements. 

Major  software  projects  such  as  AFOS  require  hundreds  of 
staff  years  of  developmental  work.  Because  this  work  is  done  in¬ 
frequently,  organizations  normally  lack  the  in-house  personnel  to 
develop  their  own  systems.  As  a  result,  it  is  normal  to  contract 
out  the  development  with  the  agency  retaining  oversight,  manage¬ 
ment,  and  approval  responsibilities.  Because  NWS  underestimated 
the  magnitude  of  AFOS  development  and  wanted  to  hold  costs  to  the 
minimum,  it  decided  to  develop  AFOS  in-house.  This  severely 
strained  NWS  resources. 

The  Director  of  NWS  indicated  that  in  developing  a  new  sys¬ 
tem  to  replace  AFOS,  greater  reliance,  where  appropriate,  would  be 
placed  on  contracting  development.  We  believe  that  contracting 
out  large  software  efforts  in  the  future  will  relieve  staffing 
constraints  and  should  result  in  higher  quality  products  because 
the  work  will  be  performed  by  full-time,  experienced  ADP  develop¬ 
ment  personnel. 
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CONCLUSIONS 


The  applications  software  developed  for  AFOS  has  serious 
problems  which  can  be  resolved  only  by  a  significant  redesign  and 
redevelopment  effort.  In  developing  the  software,  NWS  failed  to 
follow  standard  procedures.  This  deficiency  was  particularly 
evident  in  the  areas  of  documentation  and  program  testing. 

As  we  have  previously  reported  to  the  Congress,  development 
and  maintenance  of  applications  software  for  the  Government  is 
frequently  not  effective  because  management  practices  generally 
accepted  in  the  information  systems  profession  are  not  usually 
followed.  Agencies  frequently  do  not  prepare  requirements  analy¬ 
ses,  cost-benefit  studies,  or  comprehensive  project  plans,  nor  are 
full-time  project  managers  with  authority,  responsibility,  and 
accountability  always  assigned  to  software  projects. 

NWS'  problems  with  software  development  were,  in  part,  caused 
by  a  lack  of  initial  understanding  of  the  project  size  and  com¬ 
plexity.  As  a  result,  NWS  performed  work  in-house  beyond  its 
staffing  and  experience  capability.  The  software  development  ef¬ 
fort  should  have  been  contracted  out,  and  NWS  should  have  hired 
staff  members  with  the  required  training  and  experience. 

RECOMMENDATIONS 


We  recommend  that  the  Secretary  of  Commerce  direct  NWS  to 

— adhere  to  standard  software  development  practices  in 
completing  AFOS  and  in  developing  a  new  system  and 

— fully  document  the  AFOS  software  to  meet  the  needs  of 
the  developing  staff  and  operating  personnel. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

NWS  stated  that  AFOS  software  meets  all  essential  requirements 
for  field  use  and  the  original  objectives  as  listed  in  the  AFOS 
Program  Development  Plan  and  subsequently  determined  by  interaction 
with  operational  personnel.  However,  AFOS  does  not  meet  a  number 
of  NWS'  initial  requirements  prepared  in  1974.  Nor  does  it  meet 
the  updated  requirements  study  completed  in  1980,  which  stated  the 
following  system  specifications  for  AFOS  software:  (1)  it  should 
be  developed  in  a  modular  way  to  allow  ready  modifications  and/or 
expansion  of  individual  software  functions,  (2)  it  should  be  ex¬ 
pandable  not  only  to  permit  an  increase  in  the  number  of  communi¬ 
cation  lines  and  peripheral  equipment  being  serviced  but  also 
to  allow  for  an  increase  in  the  number  and  types  of  software 
functions  under  development,  as  well  as  data  from  the  automated 
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meteorological  observation  system,  and  (3)  to  be  reliable  the 
software  should  not  malfunction  more  than  once  a  year.  1/ 

NWS  stated  that  the  software  capability  has  been  reduced 
in  an  attempt  to  meet  the  specifications  for  a  reliable  and 
stable  software  system.  Moreover,  NWS  maintained  that  the  soft¬ 
ware  is  currently  performing  satisfactorily  and  that  the  software 
design  has  been  validated  by  a  systematic  NWS  test  program  initi¬ 
ated  in  early  1980.  Yet  NWS  agreed  that  AFOS  software  is  complex 
and  tightly  integrated.  We  believe  this  deficiency  is  the  funda¬ 
mental  flaw  in  the  software,  the  cause  of  most  other  software  de¬ 
ficiencies,  and  can  be  remedied  only  by  a  complete  redesign.  The 
tightly  integrated  software  will  be  a  problem  to  those  who  must 
design  subsequent  changes  in  AFOS,  a  factor  NWS  acknowledges. 
Further,  AFOS  is  not  expandable.  For  example,  it  cannot  handle 
data  from  either  the  automated  meteorological  observation  system 
or  the  radar  system.  In  addition,  the  current  reliability  level 
of  AFOS  does  not  meet  original  requirements.  In  the  March  1981 
test  of  AFOS,  the  "best"  weather  station's  system  failed  5  times 
per  day  and  the  "worst"  site  experienced  crashes  36  times  in  1  day. 
With  the  increase  in  volume  of  activity  and  products  throughout 
the  nationwide  system,  coupled  with  the  unreliable  and  untested 
system  segments,  AFOS  is  not  likely  to  meet  the  specifications  of 
one  malfunction  per  year  and  may  be  unable  to  meet  the  August- 
September  test  results  of  one  to  two  times  per  day. 

NWS'  claim  that  software  problems  have  been  corrected  and 
that  AFOS  software  works  satisfactorily  is  not  well  founded. 

The  magnitude  and  gravity  of  identified  deficiencies  cannot  be 
corrected  without  extensive  and  costly  modifications  and  a 
series  of  exhaustive  tests.  In  particular,  the  operating  sys¬ 
tem  is  not  capable  of  meeting  the  requirements  of  the  radar 
and  satellite  systems.  In  short,  the  AFOS  software  has  been 
improved  to  the  point  where  it  can  automate  the  functions  pro¬ 
vided  by  the  current  system.  The  software  remains  incapable  of 
meeting  the  new,  more  advanced  features  it  was  intended  to  pro¬ 
vide  . 


1/NWS'  original  specification  that  the  AFOS  software  should  not 
fail  more  than  once  a  year  was  identified  i-  NWS'  minimum  re¬ 
quirements  document  for  AFOS.  This  goal  is  unrealistic  con¬ 
sidering  the  state  of  today's  technology.  We  also  agree  with 
NWS'  current  contention  that  AFOS  can  fail  more  frequently 
than  one  time  a  year  without  disrupting  operations.  In  its 
August-September  1981,  validation  test,  AFOS  software  failed 
on  an  average  of  one  to  two  times  per  day  at  each  field  site. 
This  was  a  considerable  improvement  from  March  1981  when  the 
software  failed  from  5  to  32  times  per  day  at  each  site.  We 
continue  to  have  reservations  about  the  current  failure  rate 
of  one  to  two  times  per  day. 


Essentially,  NWS  divided  its  management  of  AFOS  into  two 
phases:  (1)  original  development  work  occurring  from  about  1974 

to  mid-1979  and  (2)  mid-1979  to  the  present.  During  the  first 
phase  NWS  described  its  management  approach  as  being  an  "iterative 
process."  NWS  explained  that  it  had  difficulty  determining  user 
requirements  because  forecasters  follow  variable  forecasting 
procedures  and  are  generally  unfamiliar  with  ADP.  Therefore, 

NWS  employed  an  iterative  process  to  capture  and  define  the  more 
subtle  software  requirements.  NWS  claims  that  despite  earlier 
problems,  this  process  resulted  in  an  effective  system  now  in 
actual  field  use  which  satisfactorily  performs  essential  tasks 
reliably  and  rapidly.  NWS  noted  that  this  process  often  yields 
the  most  satisfactory  systems  performance.  We  agree  that  an 
iterative  process  can  be  useful  and  even  successful.  However, 
an  effective  iterative  process  must  be  planned  and  followed  as 
such.  We  found  no  evidence  in  NWS  planning  documents  that  an 
iterative  process  was  intended  when  AFOS  development  started, 
or  is  anything  more  than  an  after-the-fact  explanation. 

The  second  management  phase  began  when  NWS  management 
determined  that  the  developers  were  having  technical  difficulty 
in  developing  and  testing  the  system  and  there  was  a  need 
to  strengthen  overall  program  management  by  initiating  certain 
development  disciplines.  According  to  NWS,  this  strengthening 
included  invoking  a  software  change  control  process,  adequately 
documenting  the  software,  and  reestablishing  a  test  program. 

NWS  stated  in  its  response  that  contractor  personnel  were 
used  in  AFOS  development.  NWS  did  hire  a  limited  number  of 
contract  programers  to  work  under  the  direction  of  NWS  managers. 
However,  this  use  of  outside  contractors  was  not  adequate  to  ad¬ 
dress  the  significant  burden  placed  on  NWS  personnel  of  develop¬ 
ing  AFOS  in  addition  to  their  regular  duties.  We  continue  to 
believe  that  NWS  should  use  contractors  to  perform  technical  work 
and  concentrate  NWS  staff  on  project  management. 

NWS  stated  that  improvements  have  been  incorporated  into 
AFOS  and  that  improvements  in  performance  and  ease  of  software 
maintenance  can  be  made  in  the  future.  We  acknowledge  the  im¬ 
provements  in  AFOS'  performance,  which  were  in  part  demonstrated 
during  the  August-September  1981  validation  test.  However,  im¬ 
provement  in  AFOS'  reliability  was  obtained  at  the  expense  of 
deleting  features  already  built  into  the  system  in  order  to  re¬ 
duce  the  strain  on  limited  computer  core  memory.  Most  of  the 
improvements  in  AFOS  were  the  substitution  of  one  capability  for 
another.  In  addition,  the  improvements  to  AFOS  have  been  limited 
to  improving  the  system's  ability  to  automate  the  functions  per¬ 
formed  by  the  current  system.  Future  improvements  to  AFOS  cannot 
be  made  to  the  extent  that  NWS  stated  in  its  response  because 
AFOS  software,  as  NWS  noted,  is  not  modular  in  construction.  If 
AFOS  is  to  be  significantly  improved,  the  system's  component  parts 
(hardware,  software,  and  telecommunications)  must  be  replaced 
entirely.  We  therefore  continue  to  have  major  reservations  about 


NWS'  intention  to  implement  AFOS  nationwide  before  a  comprehen¬ 
sive  reappraisal  is  made. 

NWS'  comments  are  summarized  in  appendix  I  and  presented 
in  their  entirety  in  appendix  II. 
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CHAPTER  5 

THE  AFOS  TELECOMMUNICATIONS  SYSTEM  IS  INADEQUATE 

The  telecommunications  system  that  is  an  integral  part  of 
AFOS  performs  very  important  functions.  It  handles  the  entry 
of  information  into  AFOS,  the  removal  and  storage  of  information, 
and  the  transmission  of  information  in  the  system.  Since  AFOS 
handles  communications  between  more  than  200  NWS  field  offices, 
the  FAA ,  the  Air  Force,  the  Navy,  and  the  NMC,  and  must  do  so 
in  a  highly  reliable  manner,  these  functions  are  critical.  Addi¬ 
tionally,  the  telecommunications  system  must  be  flexible  so  that 
other  systems  currently  under  development  by  NWS,  such  as  remote 
area  meteorological  observing  stations  and  satellite  and  radar 
data,  can  be  incorporated. 

We  found,  however,  that  the  AFOS  telecommunications  system 
is  unreliable,  inflexible,  and  expensive  to  operate  because  of 
inappropriate  design  and  poor  software.  NWS  is  aware  of  these 
problems  and  intends  to  design  a  new  telecommunications  system 
as  part  of  its  complete  new  system  but  feels  the  current  design 
will  support  AFOS  operations. 

THE  AFOS  TELECOMMUNICATIONS  DESIGN 

A  reliable  telecommunications  system  consists  of  two  basic 
components:  first,  a  telecommunications  design,  which  encom¬ 

passes  how  the  information  is  sent  (satellite  or  cable)  and 
how  the  various  users  are  connected  (each  office  connected  di¬ 
rectly  to  a  central  office  or  all  offices  connected  in  a  circuit) 
and  second,  communications  software,  which  sends,  receives,  and 
stores  the  information  in  the  minicomputer  at  each  office. 

The  telecommunications  design  that  NWS  selected  for  AFOS  is 
known  as  a  loop.  In  this  design,  WSFOs  in  each  of  NWS'  four  re¬ 
gions  are  connected  in  a  circuit  and  information  passes  in  both 
directions  around  this  circuit  to  every  WSFO.  All  four  regional 
loops  and  the  NMC  are  connected  at  the  SMCC.  WSOs  are  connected 
to  their  "parent"  WSFO  by  direct  communication  lines  similar  to 
spokes  in  a  wheel.  This  type  of  communications  design  is  called 
a  star. 

The  telecommunications  software  is  modified  specifically 
for  each  office  to  perform  various  functions.  Individual  modifi¬ 
cations  are  made  because  each  office  has  different  requirements 
for  information.  All  weather  information  passes  around  the  loop, 
and  each  office  must  extract  the  information  it  needs  to  support 
users  in  its  specific  area.  Additionally,  field  offices  must  in 
turn  relay  information  to  other  users,  such  as  private 
meteorologists . 

Additionally,  because  NWS  shares  weather  information  with 
other  services,  a  central  communications  office  that  exchanges 
this  information  is  required.  The  System  Monitoring  and 


Coordination  Center,  designed  for  this  purpose,  also  monitors 
the  entire  telecommunications  network  to  ascertain  its  status 
and  helps  offices  to  replenish  their  information  bases  in  the 
event  of  a  computer  failure. 

AFOS  telecommunications 
design  is  unreliable 

The  AFOS  loop  telecommunications  design  is  unreliable  because 
a  failure  at  any  station  in  the  loop  affects  every  other  station. 
Therefore,  overall  loop  reliability  depends  on  the  number  of  sta¬ 
tions.  Reducing  the  number  of  stations  on  a  loop  circuit  increases 
overall  reliability.  The  original  AFOS  telecommunications  design 
connected  all  WSFOs  in  one  large  national  loop.  This  design  was 
changed  into  four  smaller  regional  loops  to  increase  reliability 
and  allow  better  system  management. 

An  additional  factor  contributing  to  the  low  reliability 
of  loops  is  their  unsuitability  for  a  nationwide  common-carrier 
system  like  AFOS.  One  leading  telecommunications  consultant 
described  loops  and  their  disadvantages  as  follows: 

"Major  disadvantages  of  loops  are  their  relatively 
inferior  reliability  and  response  time  properties 
when  connecting  large  numbers  of  terminals  at 
traditional  common  carrier  speeds." 

★  *  *  *  * 

"Loops  are  useful  in  limited  distance  applications 
where  many  individual  terminal  locations  must  be 
connected  in  a  relatively  small  geographical  area." 

This  design  was  selected  because  NWS  emphasized  communica¬ 
tion  line  costs  to  the  exclusion  of  other  operational  factors 
and  characteristics.  An  advantage  of  loops  is  that  they  are  one 
of  the  cheapest  forms  of  telecommunications.  However,  the  selec¬ 
tion  has  proven  to  be  a  false  economy. 

NWS  has  added  several  features  in  an  attempt  to  offset  the 
inherent  reliability  problems  of  the  loop  architecture;  however, 
each  of  these  features  creates  an  additional  workload,  increases 
system  complexity,  and  compounds  the  chance  for  further  errors. 

The  workload  imposed  by  these  added  features  may  be  counter¬ 
productive  because  AFOS  is  at  times  very  heavily  loaded,  which 
may  result  in  further  system  instability. 

AFOS  telecommunications 
software  has  problems 

An  important  part  of  the  telecommunications  system  is  the 
communications  software  in  the  various  AFOS  minicomputers.  This 
software  controls  the  flow  and  storage  of  messages  at  field 
sites . 
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Normally,  the  communications  software  is  designed  as  an 
independent  module  so  that  changes  to  the  operating  system  and 
applications  software  do  not  affect,  or  require  modifications  to, 
the  communications  software.  This  is  especially  important  in  a 
system  like  AFOS  which  anticipates  many  changes,  such  as  the  addi¬ 
tion  of  new  capabilities.  However,  the  AFOS  communications  soft¬ 
ware  was  closely  coupled  with  the  other  software  in  an  effort  to 
conserve  core  memory.  NWS'  failure  to  design  it  as  an  independent 
module  not  only  limits  the  system's  flexibility  but  makes  the 
identification,  isolation,  and  correction  of  faults  difficult  be¬ 
cause  changes  in  one  software  system  will  often  have  unknown  ef¬ 
fects  on  the  other  software  systems. 

Because  of  this  inflexibility,  new  capabilities  cannot  easily 
be  added  to  the  system.  These  capabilities,  made  possible  by 
advances  in  technology  and  changing  user  requirements,  are  ex¬ 
pected  to  be  substantial  in  the  next  decade.  An  NWS  study  team  in 
a  January  1980  study  1/  analyzed  these  new  requirements  and  ways 
that  NWS  could  reorganize  to  meet  them.  The  study  concluded  that 
NWS  would  have  to  rely  increasingly  on  automating  weather  observa¬ 
tions  and  warning  dissemination.  This  automation  not  only  involves 
current  manual  observations  but  also  increases  the  use  of  remote 
sensing,  from  the  ground  and  satellites,  of  the  oceans  and  the 
upper  atmosphere.  Remote  sensing  greatly  increases  the  data  avail¬ 
able,  its  timeliness,  and  the  accuracy  of  forecasts.  Automating 
these  observations  would  free  personnel  for  new  tasks.  However, 
incorporating  these  capabilities  is  dependent  on  a  flexible  com¬ 
munications  system,  currently  nonexistent  in  NWS. 

The  telecommunications  software  has  other  problems. 

— The  software  is  incapable  of  transmitting  data  to  private 
meteorologists  and  other  users  of  NWS  forecasts  and  ob¬ 
servations  and  cannot  be  modified  to  perform  this  function. 

— Some  of  the  backup  capability  for  use  in  emergencies  has 
been  removed. 

— Three  systems  that  should  be  able  to  interface  with  AFOS 
beginning  in  1985  cannot  be  added  to  the  system. 

— The  use  of  these  systems,  which  are  currently  under  de¬ 
velopment,  is  dependent  on  their  being  tied  in  with 
AFOS.  (See  ch.  7.  ) 

TELECOMMUNICATIONS  SYSTEM  DOES 
NOT  MATCH  ORGANIZATIONAL  PHILOSOPHY 

The  loop  structure  of  the  AFOS  telecommunications  system 
conflicts  with  NWS'  decentra] ' "ed  management  practice  and 


1/MA  Proposal  for  the  National  Weather  Service  Field  Organization 
in  the  1980's." 
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philosophy.  NWS  field  offices  are  intended  to  exercise  a  great 
deal  of  flexibility  and  independence  to  meet  local  needs;  the 
actions  of  one  office  have  little  effect  on  others.  The  loop 
architecture,  however,  makes  field  offices  dependent  on  one 
another  in  a  way  that  causes  system  problems  at  one  station  to 
have  an  extremely  adverse  effect  on  other  stations.  For 
example,  if  one  station  does  not  take  corrective  action  follow¬ 
ing  a  software  malfunction,  a  number  of  stations  can  completely 
malfunction.  This  condition  has  frequently  occurred  in  the  past. 
In  a  few  cases  almost  an  entire  regional  loop  has  been  shut 
down  because  of  a  malfunction  at  one  site.  To  maintain  the  loop 
architecture  operationally  requires  highly  standardized  proce¬ 
dures  and  a  strong  enforcement  policy.  Because  NWS  is  not 
organized  to  manage  a  high  level  of  interdependence  and  does  not 
wish  to  impose  the  level  of  system  discipline  required  to  make 
the  system  work,  organizational  conflicts  are  created. 

TELECOMMUNICATIONS  REQUIRES 
SUBSTANTIAL  SUPPORT 

The  telecommunications  system  requires  a  high  level  of  manual 
intervention  to  operate,  for  two  reasons.  First,  inadequate  hard¬ 
ware  capacity  and  software  stability  problems  forced  NWS  to  aban¬ 
don  many  of  the  planned  automatic  features.  Second,  low  reliabil¬ 
ity  and  system  complexity  forced  officials  to  assign  a  full-time 
individual  to  monitor  the  system  at  every  field  office  in  which 
AFOS  is  installed. 

Because  of  the  high  degree  of  interdependence  of  stations  on 
the  loop,  failure  of  a  communications  line  or  station  is  of  criti¬ 
cal  importance,  and  therefore  a  number  of  recovery  features  were 
planned  for  AFOS.  They  were  of  two  types,  automatic  and  manual. 
Automatic  features  included  dialing  a  communications  line  to  re¬ 
place  a  failed  line.  NWS  has  been  forced  to  remove  most  or  the 
automatic  backup  features,  thus  requiring  greater  manual  inter¬ 
vention  for  failure  recovery  than  planned.  For  example,  if  a 
station  on  a  loop  fails,  the  failure  must  be  fixed  within  10 
minutes  or  a  bypass  switch  must  be  manually  thrown  to  cut  the 
site  out  of  the  loop.  Failure  to  do  so  can  cause  data  to  be  lost 
and  can  create  failures  in  adjoining  stations. 

The  designers  of  AFOS  assumed  the  system  would  be  so  stable 
that  it  would  practically  run  itself  without  manual  intervention. 
Based  on  this  assumption,  NWS  did  not  initially  give  operators 
needed  training  in  diagnosing  problems  and  procedures  to  effect 
recovery.  Because  of  reliability  problems,  NWS'  assumptions  about 
the  system  running  itself  have  been  inaccurate.  The  system  re¬ 
quires  a  very  knowledgeable  and  experienced  person  to  work  on  it 
full  time  at  every  office  to  maintain  operations  at  an  acceptable 
level . 
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SYSTEM  MONITORING  AND  COORDINATION 
CENTER  HAS  MANY  PROBLEMS 


Since  the  SMCC  is  the  heart  of  AFOS  communications,  the  over¬ 
all  quality  of  communications  support  is  completely  dependent  on 
this  unit's  operation.  All  four  regional  loops  and  the  NMC 
are  connected  here.  Information  from  all  sources  flowing  to  and 
from  the  NMC  passes  through  the  SMCC,  including  information  ex¬ 
changed  with  the  other  weather  services  (FAA,  Air  Force,  and  Navy). 
In  addition,  the  SMCC  monitors  the  status  of  all  offices  in  the 
system  and  helps  them  reestablish  their  data  bases  following  system 
failures . 

NWS  has  been  working  on  making  the  SMCC  operational  but  it 
still  has  many  problems.  These  problems  occurred  because  NWS 
seriously  underestimated  the  resources  needed.  Problems  such  as 
insufficient  equipment,  inadequate  software,  and  lack  of  monitor¬ 
ing  have  led  to  cancellation  of  planned  tests,  inability  to  sup¬ 
port  malfunctioning  sites,  and  serious  concerns  about  the  system's 
ability  to  support  all  four  regional  loops  if  AFOS  becomes  fully 
operational . 

NWS  originally  expected  that  the  SMCC  would  need  only  two 
minicomputers.  The  SMCC  now  has  12  AFOS  minicomputers.  More  may 
be  needed  because  the  workload  and  number  of  minicomputers  are 
largely  dependent  on  the  telecommunications  system's  reliability. 
The  lower  the  reliability,  the  more  stations  that  must  be  sup¬ 
ported  because  of  communications  failures. 

SMCC  software,  originally  developed  by  modifying  the  standard 
WSFO  software,  also  has  problems.  This  software  was  not  designed 
for  and  does  a  poor  job  of  meeting  the  SMCC's  requirements.  For 
example,  the  SMCC  software  was  not  designed  to  perform  system 
monitoring,  build  a  data  base  on  system  malfunctions,  readily  sup¬ 
port  the  SMCC  functions  of  data  base  backup  and  replenishment,  and 
act  as  a  remote  support  for  malfunctioning  sites.  This  condition 
contributes  to  the  SMCC's  current  instability.  The  SMCC  software 
still  will  not  adequately  perform  many  functions  necessary  for  the 
system  to  be  used  operationally. 

A  particularly  important  function  that  NWS  initially  neglected 
is  the  SMCC's  monitoring  function.  This  function  should  notify 
personnel  of  malfunctions  so  that  corrective  action  can  be  taken 
and  a  historical  record  of  events  can  be  built  for  planning  pur¬ 
poses.  Recent  network  tests  have  shown  the  importance  of  monitor¬ 
ing  and  that  the  SMCC  lacks  this  fundamental  capability.  Because 
these  functions  were  assigned  a  low  priority,  AFOS  project  man¬ 
agers  did  not  provide  adequate  staff  to  develop  them. 

NWS  assigned  the  SMCC  a  low  priority  because  the  original 
assumptions  in  developing  the  system  included  a  high  level  of 
system  reliability.  Had  this  reliability  been  achieved,  the 
SMCC's  monitoring  function  would  not  have  been  as  important 
as  it  has  become. 
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Concerning  the  SMCC,  in  March  1981  NWS  stated: 

"Among  the  largest  remaining  uncertainties  about  the 
AFOS  system  are  the  adequacy  of  the  design  and  per¬ 
formance  of  the  SMCC.  The  questions  are  compounded 
by  the  increased  demands  of  full  national  operations 
with  four  regional  loops,  several  spurs,  network 
monitoring  and  backup  functions.  *  *  *  it  is  clear 
that  a  carefully  planned  effort,  probably  substantial, 
is  required  before  a  sufficient  level  of  confidence 
in  the  readiness  of  SMCC  can  be  achieved." 

THE  NEED  FOR  TELECOMMUNICATIONS  ALTERNATIVES 

NWS  recognizes  the  importance  of  a  reliable  telecommunica¬ 
tions  system.  Following  tests  conducted  in  January  1981,  NWS 
program  developers  reported  to  management  that: 

"The  communication  system  must  be  made  more  reliable. 

The  high  rate  of  line  failures  must  be  eliminated  and 
the  source  of  communications  deficiencies  that  cause 
AFOS  crashes  must  be  found  and  corrected.  *  *  * 

Failure  to  solve  it  will  cause  AFOS  to  fail." 

***** 

"The  communications  design  is  deficient  in  that  a 
station  may  destroy  loop  integrity  by  merely  failing 
to  go  into  by-pass  during  periods  of  computer  outage. 

This  is  compounded  by  the  inability  of  SMCC  to  know 
what  stations  are  causing  this  problem." 

NWS  has  acknowledged  that  the  system  places  "added  demands 
for  network  monitoring  and  management  discipline,  configuration 
control,  and  orchestrated  implementation  of  changes."  Further, 
because  of  the  importance  of  telecommunications  for  meeting 
current  and  future  needs,  NWS  intends  to  replace  the  telecommu¬ 
nications  system  as  part  of  its  new  system  development. 

NWS  now  acknowledges  that  the  loop  architecture  is  not  a 
viable  long-term  approach  and  has  contracted  with  a  consulting 
firm  for  a  comprehensive  analysis  of  potential  alternatives  for 
use  in  the  new  system.  This  analysis  also  includes  the  potential 
impact  of  requirements  outside  AFOS,  as  well  as  the  management 
implications  of  alternatives.  The  first  part  of  this  analysis 
has  been  completed.  The  consulting  firm  has  made  the  following 
points  about  any  future  telecommunications  architecture: 

— It  must  be  very  flexible  to  expand  and  contract  with 
future  needs.  This  flexibility  is  especially  important 
because  firm  data  on  future  requirements  is  lacking. 

— It  is  critical  that  the  architecture  selected  require 
minimal  management  involvement. 
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CONCLUSIONS 


The  AFOS  telecommunications  system  is  not  appropriate  for 
meeting  NWS'  needs.  It  requires  extensive  and  costly  onsite 
staff  support.  The  system  is  inflexible  and  unreliable  and  con¬ 
flicts  with  NWS'  organizational  philosophy.  The  telecommunica¬ 
tions  system  will  need  to  be  completely  replaced,  and  in  replac¬ 
ing  the  loop  system,  NWS  should  adopt  an  alternative  which  is 
compatible  with  its  organizational  structure. 

NWS'  decision  to  use  the  loop  approach  was  based  on  a  desire 
to  hold  the  system's  cost  to  a  minimum.  Reliability  in  telecom¬ 
munications  is  closely  related  to  cost.  Loops  are  one  of,  if  not 
the  least  expensive  form  of  telecommunications  in  terms  of  pure 
communications  cost.  However,  the  tradeoff  is  lower  reliability 
and  less  flexibility. 

RECOMMENDATION 

We  recommend  that  the  Secretary  of  Commerce  direct  NWS  to 
replace  the  current  telecommunications  system  as  part  of  its 
development  of  a  new  system. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

NWS  stated  that  the  telecommunications  system  is  flexible 
and  reliable  and  that  it  does  not  require  a  high  level  of  manual 
intervention  because  of  unstable  software  and  inadequate  hardware 
memory  capacity. 

It  is  important  to  note  that  a  1980  assessment  of  the  AFOS 
loop  architecture  disclosed  that  the  reliability  of  the  AFOS 
telecommunications  system  continues  to  be  a  major  problem.  Fur¬ 
ther,  a  March  1981  test  of  AFOS  showed  that  system  performance 
was  poor  and  that  AFOS  was  inadequate  for  operational  use.  It 
did  not  perform  according  to  specifications,  and  it  does  not  meet 
the  agency's  overall  requirements.  NWS  is  currently  evaluating 
alternatives  and  has  indicated  that  it  plans  to  develop  a  replace 
ment  telecommunications  architecture  as  part  of  a  new  system. 

NWS  disagreed  that  AFOS  requires  a  high  level  of  personnel 
support.  NWS  used  an  inappropriate  measure — the  actual  time 
to  deal  with  a  specific  problem — as  the  basis  for  the  statement. 
This  measure  is  only  part  of  the  calculation  of  overall  support 
staff  needs.  At  each  site,  at  least  50  percent  of  one  person's 
time  is  devoted  to  training  personnel  to  resolve  technical  prob¬ 
lems.  This  is  a  high  level  of  investment  in  such  resources.  NWS 
has  established  a  new  software  diagnostic  course  for  AFOS  mainten 
ance  personnel,  primarily  so  that  they  can  diagnose  the  cause  of 
software  crashes.  NWS  has  established  AFOS  coordination  groups 
of  about  10  full-time  people  in  each  region. 
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Operating  AFOS  requires  many  more  people  than  are  required 
to  operate  and  maintain  the  current  system.  This  became  evident 
as  NWS  added  119  positions  for  the  AFOS  project.  NWS  contends 
that  AFOS  has  resulted  in  over  200  fewer  positions  to  support 
the  NO A A  Weather  Wire  Service.  This  NWS  statement  is  misleading 
because  it  refers  to  authorized  personnel  ceilings,  which  have 
never  been  filled,  and  not  to  people  on  NWS'  payroll.  In  fact, 
NWS  has  been  supporting  the  Weather  Wire  Service  without  these 
positions  for  years  using  the  current  system,  not  AFOS. 

NWS  should  continue  examining  the  need  to  replace  the  AFOS 
telecommunications  system  as  part  of  the  new  system  proposed 
for  development.  A  new  telecommunications  system  would  provide 
capability  that  the  agency  requires  but  cannot  receive  from 
the  current  loop  architecture  because  of  design  and  operational 
limitations.  Further,  a  new  telecommunications  system  would 
improve  reliability  and  minimize  failures  in  the  telecommunica¬ 
tions  network,  which  is  crucial  to  the  operation  of  the  total 
system.  More  recent  tests  have  shown  improvements,  but  NWS 
recognizes  the  limitations  of  loops  and  is  studying  alternative 
systems . 

NWS  also  stated  that  the  SMCC  is  adequately  performing 
its  functions.  We  note  that  NWS  has  apparently  completed  and 
is  using  network  monitoring  software  at  the  SMCC.  We  have  not 
verified  or  reviewed  this  information  but  believe  it  is  accurate. 
This  improvement  should  help  NWS  to  monitor  the  AFOS  network 
and  should  be  of  considerable  assistance  to  the  SMCC.  However, 
since  only  selected  segments  of  the  SMCC  functions  and  associated 
software  have  been  tested,  untested  segments  remain.  NWS  has 
little  or  no  assurance  that  they  meet  requirements.  Tests  of 
the  SMCC's  software  in  terms  of  a  fully  loaded  system  would 
be  necessary  to  assure  system  stability.  Further,  in  performing 
its  monitoring  function,  the  SMCC  must  demonstrate  its  ability 
to  adequately  respond  to  system  failures  along  the  telecommunica¬ 
tions  network.  Before  AFOS  is  implemented,  the  SMCC  operation 
and  associated  functions  demand  the  successful  completion  of 
comprehensive  tests. 

NWS'  comments  are  summarized  in  appendix  I  and  presented  in 
their  entirety  in  appendix  II. 


CHAPTER  6 


THE  AFOS  HARDWARE  NEEDS  TO  BE  REPLACED 


The  hardware  in  the  AFOS  system  includes  310  minicomputers. 
Each  WSFO  has  two  minicomputers  so  that  if  one  fails,  the  other 
can  provide  backup.  Each  WSO  in  the  AFOS  network  has  one  mini¬ 
computer.  In  addition  to  minicomputers,  each  field  office  has 
a  variable  number  of  work  stations  for  meteorologists.  Also 
included  is  a  control  console  for  maintaining  the  operation 
of  the  station's  equipment.  The  operation  of  this  hardware 
and  the  functions  it  performs  are  controlled  by  the  operating 
system,  a  software  component.  The  operating  system  is  critical 
to  the  effective  use  of  the  system  and  controls  the  functions 
of  all  hardware  components. 

NWS  has  experienced  serious  problems  with  the  operation 
and  maintenance  of  its  computers,  the  system's  hardware,  and 
the  operating  system.  Further,  the  hardware  does  not  adequately 
meet  agency  requirements.  We  are  also  concerned  that  NWS'  logis¬ 
tics  program  for  maintaining  spare  parts  may  not  be  adequate  to 
support  a  fully  operational  system. 

AFOS  HARDWARE  HAS  PROBLEMS 

Two  major  problems  prevent  the  hardware  from  meeting  NWS' 
requirements.  First,  the  performance  of  the  hardware's  operating 
system  falls  far  short  of  its  intended  role  in  directing  and 
controlling  the  computer.  Second,  the  hardware  has  inadequate 
core  memory  capacity  to  store  software  and  forecast  data.  In 
our  opinion,  correcting  these  problems  of  the  current  computers 
would  not  be  cost  effective.  As  a  result,  NWS  should  replace 
the  hardware  as  part  of  a  redesign  and  development  effort. 

Operating  system  is  inadequate 

An  operating  system  is  highly  specialized  software  developed 
by  the  system's  manufacturer.  It  enables  the  hardware  and  appli¬ 
cations  software  to  work  together  in  accomplishing  user  needs. 

The  AFOS  operating  system  controls  each  minicomputer's  central 
processing  unit.  It  is  integral  to  the  operation  of  the  mini¬ 
computers  and  is  required  for  operating  the  hardware.  The  AFOS 
operating  system  functions  much  like  the  human  brain  in  that  it 
controls  the  independent  parts  of  the  system  and  causes  them  to 
act  as  a  unified  whole.  Because  of  the  specialized  experience 
required  to  develop  operating  systems,  they  are  usually  built 
and  modified  by  equipment  manufacturers  or  software  vendors.  As 
a  rule,  user  organizations  do  not  modify  manufacturers'  operating 
systems . 

NWS  requires  an  operating  system  which  can  concurrently  per¬ 
form  a  number  of  separate  functions.  The  operating  system  ac¬ 
quired  with  the  hardware  is  unable  to  perform  these  functions. 
When  it  purchased  the  AFOS  minicomputers,  NWS  was  aware  of  the 
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operating  system’s  limitations.  However,  at  the  time  of  the 
purchase,  minicomputer  operating  systems  which  met  NWS'  require¬ 
ments  were  not  available.  Minicomputers  were  a  relatively  new 
product,  and  NWS  believed  the  operating  system  could  be  modified 
to  meet  its  needs. 

It  should  be  noted  that  NWS  made  every  effort  to  obtain 
an  appropriate  operating  system.  NWS  initially  found  a  vendor 
who  agreed  to  provide  a  minicomputer  and  develop  an  operating 
system  to  meet  its  needs.  Unfortunately,  the  vendor  was  unable 
to  provide  it,  and  NWS  decided  to  go  with  its  second  choice — 
the  present  minicomputer  and  operating  system.  NWS  then  modified 
the  operating  system  to  meet  its  requirements. 

These  modifications  to  the  AFOS  operating  system  were  made 
contrary  to  the  advice  of  independent  contractors.  Further,  the 
changes  were  poorly  documented  and  their  effects  not  fully 
understood.  The  following  problems  have  resulted  from  these 
modifications : 

— Maintaining  the  minicomputers  will  be  difficult  and 
expensive . 

— The  system's  complexity  has  been  increased  because 
the  applications,  telecommunications,  and  operating 
system  software  have  been  tightly  integrated.  When 
any  one  of  these  is  modified,  changes  to  the  other 
two  are  often  required.  This  situation  increases 
software  costs  and  greatly  reduces  flexibility  in 
modifying  AFOS. 

— Hardware  and  software  improvements  made  by  the  mini¬ 
computers'  manufacturer  cannot  be  incorporated  into 
the  system  without  considerable  cost  and  effort. 

NWS  has  been  making  a  substantial  effort  to  resolve  the 
minicomputers'  operating  system  deficiencies  for  several  years 
without  finding  an  adequate  solution.  Not  only  has  it  modified 
the  operating  system  extensively,  but  it  has  continued  its  at¬ 
tempts  to  procure  an  operating  system  suitable  for  the  current 
minicomputer.  As  time  passes  the  operating  system  problem  be¬ 
comes  more  critical.  For  example,  the  computer's  design  will 
be  10  years  old  in  1984,  the  current  target  date  for  AFOS  to  be 
a  fully  operational  system.  Further,  the  original  design  of  AFOS 
has  not  been  updated  to  incorporate  the  significant  technological 
advances  that  are  available  from  the  vendor  that  would  help  solve 
the  system's  problems  and  meet  overall  needs.  Further,  the  cost 
of  adding  adequate  capacity,  redeveloping  the  applications  and 
telecommunications  software,  and  procuring  a  new  operating  sys¬ 
tem  for  310  minicomputers  would  probably  approach  the  cost  of  an 
entirely  new  system. 
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NWS  did  not  adequately  determine  its  requirements  for  AFOS 
and  as  a  result  purchased  hardware  with  inadequate  core  memory 
capacity.  The  specific  amount  of  the  shortage  cannot  be  deter¬ 
mined  because  NWS  does  not  know  (1)  how  much  capacity  it  needs 
or  (2)  the  current  capacity  of  its  hardware.  NWS  has  concluded, 
however,  that  all  available  capacity  is  fully  utilized.  This 
shortage  of  capacity  created  problems  during  development  of  the 
system,  and  as  a  result: 

— The  AFOS  software  could  not  be  developed  as  inde¬ 
pendent  modules  and  had  to  be  tightly  integrated  to 
conserve  memory. 

— The  smaller  field  offices  (WSOs)  with  extensive  com¬ 
munication  requirements  have  limited  ability  to  per¬ 
form  onsite  data  processing,  a  planned  capability  of 
the  system. 

— Large  field  sites  (WSFOs)  lack  sufficient  capacity 
to  support  external  users. 

— Much  of  the  system's  disaster  recovery  and  backup 
capability  has  been  removed  because  of  the  core 
memory  capacity  problem. 

— The  system  cannot  be  expanded  to  meet  current  and 
future  needs  without  a  costly  addition  of  more 
storage  capacity. 

AFOS  REPAIR  COSTS  ARE  LIKELY  TO  BE  EXCESSIVE 

In  addition  to  the  minicomputers'  performance  problems, 
maintaining  them  will  be  difficult  and  costly.  System  mainte¬ 
nance  costs  will  be  higher  than  anticipated  because  NWS  did  not 
select  an  appropriate  logistics  approach  to  maintaining  a  spare 
parts  inventory. 

A  hardware  logistics  program  is  usually  implemented  under 
two  basic  approaches:  (1)  buy  all  identical  hardware  systems  and 
spare  parts  required  for  the  system's  entire  life  cycle  or  (2) 
buy  enough  parts  to  meet  initial  needs,  but  continually  update 
and  maintain  all  units  to  make  them  identical  to  the  most  current 
equipment  the  manufacturer  is  producing.  NWS  did  not  follow 
either  approach.  Instead,  it  bought  identical  new  systems  and 
the  engineering  drawings  and  specifications,  as  well  as  a  limited 
inventory  of  spare  parts,  in  the  initial  procurement.  NWS  planned 
to  have  spares  made  as  needed  to  preserve  the  system's  original 
specifications,  rather  than  to  update  them.  As  a  result,  main¬ 
taining  AFOS  will  be  expensive,  and  the  hardware  cannot  easily 
be  updated  to  take  advantage  of  new  technology  that  would  correct 
major  computer  deficiencies. 


NWS'  decision  not  to  include  ample  spare  parts  with  the 
original  computer  system  purchase  order  has  caused  problems 
with  the  availability  of  critical  parts.  Since  the  manufacturer 
no  longer  makes  many  critical  AFOS  parts  on  a  normal  basis, 
special  production  runs  for  these  parts  will  be  required.  In 
some  cases  NWS  will  need  to  find  new  subcontractors  to  produce 
parts  that  are  no  longer  available  from  the  original  source. 

Because  all  parts  must  be  produced  on  a  special  basis,  NWS  will 
be  forced  to  pay  a  premium  price.  Further,  the  premium  price  and 
the  number  of  needed  spares  are  expected  to  increase  significantly 
over  time. 

Because  NWS  did  not  follow  the  second  approach  of  updating 
and  maintaining  the  AFOS  hardware  to  make  it  identical  with  the 
manufacturer's  current  equipment,  AFOS'  capability  and  effective¬ 
ness  have  been  limited.  Generally,  computer  manufacturers  intro¬ 
duce  enhancements  in  computers  they  currently  produce  that  improve 
reliability  and  lower  operating  costs.  The  manufacturer  of  the 
AFOS  computer  made  numerous  improvements  to  the  computer's  memory 
boards  and  other  computer  components  it  currently  builds.  Ninety 
percent  of  these  changes  could  be  considered  enhancements  that 
could  have  significantly  improved  AFOS.  Nonetheless,  NWS  has  not 
been  able  to  take  advantage  of  these  modifications  and  enhancements. 

Modifying  the  AFOS  hardware  to  include  such  enhancements  and 
to  make  AFOS  identical  to  the  manufacturer's  current  equipment 
may  not  be  feasible  at  this  time.  Because  NWS  has  made  many 
changes  to  its  AFOS  equipment  and  these  changes  are  undocumented, 

NWS  cannot  determine  with  any  degree  of  accuracy  the  extent  to 
which  AFOS  equipment  differs  from  the  manufacturer's  current 
equipment.  Further,  there  is  no  assurance  that  the  AFOS  software 
will  effectively  and  efficiently  operate  at  the  most  current  re¬ 
vision  level  of  the  equipment  the  manufacturer  is  currently 
producing . 

NWS'  difficulties  in  keeping  up  with  the  manufacturer's  cur¬ 
rent  revision  level  for  its  computers  resulted  in  part  from  the 
prolonged  delays  in  completing  AFOS  and  the  rapid  pace  of  techno¬ 
logical  changes  in  minicomputers.  If  AFOS  had  been  completed  on 
schedule  in  1979,  all  the  hardware  it  needed  might  have  been  pur¬ 
chased  by  that  date. 

NWS'  MAINTENANCE  AND  LOGISTICS 
SYSTEM  WILL  NOT  SUPPORT  AFOS 

NWS'  present  maintenance  and  logistics  system  may  be  unable 
to  support  AFOS  when  the  system  is  fully  operational.  NWS  has 
already  encountered  problems  with  insufficient  spares,  late  de¬ 
liveries  of  spares,  and  extensive  repair  time.  Additionally,  the 
NWS  repair  center  has  had  hardware  maintenance  problems  that  were 
caused  by  vacant  staff  positions,  lack  of  adequate  tools,  faulty 
automatic  test  equipment,  inadequate  software,  and  inadequate 
skills  to  maintain  the  test  equipment. 
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A  key  to  effective  maintenance  capability  is  the  use  of 
automatic  test  equipment.  This  equipment's  purpose  is  to 
minimize  repair  time,  which  reduces  the  need  for  a  large  in¬ 
ventory  of  spare  parts.  However,  the  benefits  of  the  test 
equipment  have  not  been  realized  due  to  testing  program  errors 
and  the  inability  to  maintain  the  test  equipment  in  good  working 
order.  The  repair  center  estimates  that  50  percent  of  the  test 
programs  are  defective. 

Many  problems  with  logistics  would  not  have  occurred 
had  the  automatic  test  equipment  functioned  as  planned.  To  hold 
down  costs,  NWS  pared  its  spare  parts  inventory  to  the  minimum. 

It  believed  that  the  test  equipment,  coupled  with  organizational 
flexibility,  would  save  money  and  meet  requirements  for  spare 
parts.  This  approach  has  held  down  costs,  but  because  of  test 
equipment  problems,  it  may  be  unable  to  meet  NWS'  needs. 

The  problems  NWS  has  experienced  with  its  logistics  and 
maintenance  system  could  jeopardize  full  implementation  of  AFOS. 
For  example,  field  offices  have  experienced  outages  of  up  to  12 
days  due  to  a  single  parts  failure,  resulting  in  severe  reduc¬ 
tions  in  service  and  an  inability  to  meet  mission  needs. 

CONCLUSIONS 

The  AFOS  hardware  is  inadequate  for  meeting  NWS'  require¬ 
ments.  The  system's  primary  problems  are  insufficient  computer 
capacity  and  an  operating  system  lacking  the  required  level  of 
capability . 

NWS  and  independent  contractors  have  concluded  that  the 
operating  system  must  be  replaced,  and  NWS  intends  to  do  so  as 
part  of  developing  a  new  system.  Replacing  the  operating  system 
will  require  NWS  to  rewrite  its  application  software  and  to 
obtain  additional  core  memory  for  the  computers.  We  believe  the 
most  cost-effective  solution  to  these  problems  is  a  complete 
replacement  of  the  computer  hardware  and  its  operating  system. 

The  AFOS  logistic  support  system  is  likely  to  encounter 
problems  if  AFOS  is  fully  implemented.  The  primary  logistics 
problem  resulted  from  NWS'  decision  not  to  maintain  the  equipment 
at  the  manufacturer's  current  revision  levels,  in  replacing 
the  AFOS  hardware,  NWS  should  acquire  equipment  and  an  operating 
system  at  the  most  current  level  produced  by  the  manufacturer 
and  should  maintain  them  at  the  manufacturer's  current  revision 
level . 

RECOMMENDATION 

We  recommend  that  the  Secretary  of  Commerce  direct  NWS  to 
replace  the  AFOS  hardware  as  part  of  developing  a  new,  more  ad¬ 
vanced  system. 
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AGENCY  COMMENTS  AND  OUR  EVALUATION 


NWS  objected  to  our  recommendation  that  AFOS  hardware  be 
replaced  as  part  of  a  complete  system  redesign  and  disagreed 
with  our  conclusion  that  the  design  and  capabilities  of  the 
AFOS  hardware  and  operating  system  are  inadequate.  NWS  stated 
that  the  system  has  more  than  adequate  capacity  and  that  the 
hardware's  operating  system  is  completely  satisfactory  for 
NWS'  needs. 

In  its  comments  NWS  said  it  plans  to  purchase  additional 
core  memory  for  the  AFOS  hardware.  Since  memory  is  in  fact  the 
capacity  lacking  in  AFOS  hardware,  it  is  inconsistent  to  state 
that  the  capacity  is  satisfactory  and  yet  plan  to  purchase 
more  memory.  To  further  support  the  need  for  additional  capac¬ 
ity,  in  March  1981  a  major  NWS  evaluation  of  AFOS  concluded 
that  the  system  requires  at  least  $4  million  in  additional 
core  memory.  Also,  in  1981  the  AFOS  project  manager  concluded 
that  a  number  of  features  needed  to  meet  minimal  operational 
requirements  could  not  be  added  because  of  inadequate  capacity. 

We  contend  that  NWS  should  procure  additional  computer  memory 
if  AFOS  is  to  be  the  primary  system.  This  additional  computer 
storage  would  relieve  some  of  th_  strain  on  capacity. 

This  $4  million  in  added  core  memory  capacity  will  improve 
only  the  system's  ability  to  automate  the  functions  provided  by 
the  current  system.  AFOS  will  remain  unable  to  provide  the 
additional  services  and  performance  improvements  which  were 
originally  intended  for  the  system.  NWS  is  aware  of  this  and 
is  planning  to  develop  a  new  system  to  provide  this  additional 
support . 

In  1974,  the  NWS  technical  review  team  which  approved  the 
operating  system  purchased  by  NWS  concluded  that  the  operating 
system  was  inadequate.  Since  then  NWS  has  contracted  for  a  num¬ 
ber  of  independent  technical  reviews  which  have  concluded  that 
the  operating  system  is  inadequate.  In  fact,  NWS  itself  con¬ 
cluded  in  March  1981  that  the  AFOS  operating  system  is  inadequate 
and  placed  a  top  priority  on  developing,  contracting  for  develop¬ 
ment  of,  or  purchasing  an  operating  system. 

NWS  stated  that  "the  problems  with  the  flow  and  avail¬ 
ability  of  spare  parts  have  been  resolved.  However,  previous 
assessments  of  the  system  have  shown  significant  problems  with 
the  spare  parts  replacement  program."  In  the  last  year  AFOS 
sites  have  been  closed  down  for  as  long  as  4  to  6  weeks  because 
of  a  single  parts  failure.  In  an  earlier  test,  one  site  had  to 
remain  in  a  degraded  operation  for  over  48  hours  due  to  the 
parts  problem.  This  performance  level  was  initially  unaccept¬ 
able  for  minimal  operational  needs,  and  such  problems  resulted  in 


55 


L 


aaaa < 


NWS'  curtailing  AFOS'  performance  criteria.  In  this  context,  NWS 
has  significantly  increased  the  acceptable  time  allowed  to  repair 
faulty  equipment  and  for  a  site's  system  to  remain  inoperative. 

NWS'  comments  are  summarized  in  appendix  I  and  presented 
in  their  entirety  in  appendix  II. 
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CHAPTER  7 


ALTERNATIVE  APPROACHES  TO  IMPLEMENTING  AFOS 


NWS  agrees  with  our  view  that  AFOS  should  eventually  be  re¬ 
placed  and  intends  to  prepare  a  long-range  plan  for  developing  a 
new  system.  In  the  meantime,  however,  the  agency  plans  to  imple¬ 
ment  AFOS  nationally  on  an  "as  is"  basis  to  derive  limited  bene¬ 
fits  from  the  work  performed. 

AFOS  should  not  be  implemented  nationally  without  a  thorough 
analysis  of  whether  the  system  can  produce  benefits  commensurate 
with  its  costs.  AFOS  should  be  required  to  undergo  a  major  re¬ 
appraisal  before  it  is  implemented  on  a  national  basis.  This  ap¬ 
praisal  should  include  a  study  of  relative  benefits  and  costs  as 
well  as  additional  operational  testing.  We  recommend  such  an 
appraisal  because  of  evidence  that  the  system  as  designed  does 
not  meet  NWS'  original  performance  requirements,  will  be  expen¬ 
sive  to  maintain  and  operate,  and  will  ultimately  have  to  be 
replaced  by  a  new  system  because  of  its  limited  capabilities. 

An  alternative  to  full  implementation  of  AFOS  is  to  concen¬ 
trate  resources  on  designing  and  developing  a  new  system,  using 
currently  available  off-the-shelf  technology,  that  will  come 
closer  to  meeting  NWS'  needs. 

CURRENT  NWS  PLANS 

Following  a  detailed  technical  review  of  AFOS,  NWS  concluded 
in  March  1981  that  the  possibility  exists  that  the  AFOS  system 
is  not  viable.  1/  NWS  decided  to  correct  a  number  of  identified 
system  problems  and  to  conduct  an  AFOS  validation  and  demonstra¬ 
tion  test  in  August-September  1981.  The  results  of  this  test 
would  determine  whether  NWS  would  proceed  to  implement  AFOS 
nationwide  or  discontinue  all  use  of  the  system. 

NWS  conducted  the  AFOS  validation  test  as  scheduled  and 
determined  that  the  results  were  sufficiently  positive  to  justify 
proceeding  with  national  implementation  of  the  system.  NWS  cur¬ 
rently  plans  to  implement  AFOS  by  the  end  of  1982  in  all  four 
contiguous  U.S.  regions.  This  plan  calls  for  leaving  the  cur¬ 
rent  system  in  place  until  1984  as  a  backup  to  AFOS  in  the 
event  of  system  failures. 

In  implementing  AFOS,  NWS  plans  to  freeze  the  system's 
software  and  operating  procedures  to  the  greatest  extent  pos¬ 
sible.  NWS  has  stressed  that  all  changes  to  the  system  would  be 
minor  and  would  be  made  in  the  most  controlled  manner  possible. 
This  approach  should  help  stabilize  the  AFOS  system.  NWS  also 


1/NWS  Director's  memorandum  to  regional  and  office  directors 
regarding  "AFOS  National  Implementation  Decision,"  March  19, 
1981. 


plans  to  purchase  additional  core  memory  for  the  system  at  a 
cost  of  about  $4  million. 

NWS  has  also  initiated  plans  for  a  new  system  to  replace 
AFOS.  This  system ,  if  approved  and  funded  by  the  Congress, 
would  be  completed  around  1989.  NWS  currently  intends  to  develop 
the  new  system  based  on  a  new  design  rather  than  the  AFOS  design. 
The  NWS  decision  to  develop  the  new  system  totally  apart  from 
AFOS  reflects  a  broader  decision  to  defer  correcting  a  number  of 
AFOS'  technical  deficiencies.  The  new  system  is  intended  to  re¬ 
place  the  current  communications  as  well  as  AFOS. 

NWS’  EFFORTS  TO  ADDRESS 
AFOS’  LIMITATIONS 

NWS  has  addressed  AFOS'  limitations  by  correcting  problems 
in  ways  that  produce  short-term  solutions  rather  than  long-term 
improvements  that  would  assure  effective  and  efficient  operations 
over  the  next  8  to  10  years.  Results  from  various  AFOS  opera¬ 
tional  tests,  including  the  most  recent  test,  indicate  that  AFOS' 
limitations  affecting  the  long  term  will  continue  because  of 
inherent  design  constraints  that  preclude  it  from  meeting  the 
agency's  requirements.  For  instance,  to  meet  changing  require¬ 
ments  over  the  next  several  years,  the  AFOS  system  must  be 
flexible  to  the  extent  that  its  hardware,  software,  and  telecom¬ 
munications  components  can  be  effectively  and  efficiently  modified. 

Interim  steps  to  purchase  additional  core  memory  and  to  up¬ 
grade  the  AFOS  telecommunications  system,  which  is  currently 
under  assessment  by  a  consultant,  could  increase  flexibility. 
However,  the  system  still  would  not  have  sufficient  flexibility 
because  of  the  constraints  in  the  software  for  processing  user 
applications  and  performing  prescribed  functions.  The  software 
design  was  not  structured  under  a  modular  design  concept;  it  was 
poorly  documented  and  tested  on  a  limited  basis.  The  software 
limitations  are  essentially  imbedded  in  the  software  design,  and 
they  preclude  modifying  the  software  effectively,  economically, 
and  efficiently.  Consequently,  this  aspect  of  the  software  limits 
the  flexibility  of  the  entire  system. 

To  increase  the  system's  flexibility,  NWS  also  attempted  to 
modify  the  operating  system.  (This  software  executes,  directs, 
and  controls  the  computer  system's  hardware  and  system  functions 
and  processes.)  This  modification  resulted  in  improved  perform¬ 
ance  but  did  not  succeed  in  increasing  the  flexibility  to  the 
extent  required.  Rather  than  taking  further  steps  to  address 
the  problem  of  flexibility,  NWS  has  now  decided  to  forego  software 
changes  designed  to  increase  flexibility  by  freezing  all  software 
design  and  development  specifications.  Freezing  software  essen¬ 
tially  resolves  the  more  immediate  and  critical  problem  of  minimiz¬ 
ing  software  malfunctions  and  system  failures.  Yet,  freezing  the 
desiqn  and  halting  basic  design  changes  in  software  means  that  AFOS 
will  continue  to  be  an  inflexible  system,  not  capable  of  meeting 
prescribed  requirements. 


NWS*  attempt  to  address  another  AFOS  limitation — the 
capability  of  handling  future  programs  and  systems,  such  as  the 
weather  observing  satellite  program,  the  new  weather  radar  sys¬ 
tems,  and  automated  weather  observing  systems  currently  under 
development — has  not  met  with  success.  These  programs  are 
critically  important  for  improving  NWS1  ability  to  forecast 
weather.  AFOS1  inability  to  meet  future  needs  will  become  acute 
as  the  new  radar  and  satellite  systems  become  operational  about 
1985  to  1987. 

Forecasting  ability  is  often  hindered  by  a  lack  of 
meteorological  observations.  Obtaining  data  from  remote  areas 
is  difficult,  and  NWS  forecasters  must  frequently  do  without 
this  information.  To  help  overcome  this  deficiency,  NWS  is  cur¬ 
rently  conducting  experiments  in  which  automated  weather  observing 
systems  are  established  in  remote  areas  to  automatically  relay 
data  to  a  minicomputer  for  analysis. 

During  our  visit  with  NWS  officials  in  charge  of  one  of 
these  projects,  we  noted  that  the  project  demonstrated  the  poten¬ 
tial  to  save  millions  of  dollars  in  the  areas  of  improved  water 
resource  management  and  flood  control.  Thus,  remote  observation 
technology  could  bring  about  significant  improvement  in  weather 
prediction  and  millions  of  dollars  in  savings  to  the  public 
and  industry.  However,  to  translate  this  technology  from  an 
experimental  to  an  operational  state  requires  a  communications 
system  with  the  ability  to  receive  and  analyze  this  data.  Be¬ 
cause  AFOS  cannot  handle  these  re w  systems  now  being  developed, 
NWS'  ability  to  realize  the  bonifies  of  the  new  technologies 
will  be  significantly  reduced. 

THE  AFOS  VALIDATION  TEST 


We  monitored  the  August-September  1981  AFOS  validation 
test.  While  we  have  reservations  about  the  design  of  the  test, 
addressed  below,  the  following  limited  conclusions  can  be  drawn 
from  the  test:  (1)  AFOS*  performance  has  improved  in  the  areas 
of  stability,  frequency  of  "crashes,”  user  response  time,  and 
backup  systems  and  (2)  its  ability  to  support  the  Weather  Wire 
Service,  a  critical  requirement,  has  also  improved.  Also,  dur¬ 
ing  the  test  we  noted  that  AFOS  operating  personnel  seemed 
more  familiar  with  the  system  and  better  trained  than  they  were 
in  the  spring  of  1981  when  we  observed  the  earlier  AFOS  test.  It 
appears  that  increased  management  involvement,  NWS  corrections 
to  known  SMCC  problems,  emphasis  on  standard  operating  procedures, 
and  a  significant  increase  in  system  discipline  were  largely 
responsible  for  the  most  recent  improvements. 

However,  we  did  note  a  number  of  negative  results.  For 
example,  the  frequency  of  system  failure  has  remained  high  in 
relation  to  NWS*  original  standards  for  reliability.  But  re¬ 
covery  procedures  worked  well,  and  the  elapsed  time  to  recover 
was  less  than  that  previously  noted.  Also,  we  observed  that  NWS 
performed  minimal  testing  of  certain  features  and  functions 
believed  to  be  the  sources  of  potential  problems.  This  limited 
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use  of  "troublesome"  features  would  not  be  acceptable  if  AFOS  were 
operational  throughout  the  Nation  and  the  system  were  fully  loaded 
Another  concern  was  NWS'  extensive  use  of  personnel  at  all  levels 
of  management  and  staff  during  the  test.  This  intensive  support 
was  maintained  for  a  2-  to  4-week  test  and  evaluation  period,  but 
it  is  unlikely  that  NWS  could  maintain  this  level  of  support  on 
a  day-to-day  basis  over  the  next  several  years  of  AFOS  operation. 

Test  limitations 

The  AFOS  validation  test  conducted  by  NWS  in  August- 
September  1981  had  deficiencies  in  its  design  as  well  as  in  the 
results  achieved.  The  results  of  this  test  therefore  do  not  pro¬ 
vide  a  sufficient  basis  for  a  decision  to  implement  AFOS  nation¬ 
wide  . 

The  test  was  limited  in  a  number  of  respects.  First,  it 
tested  AFOS'  ability  to  handle  only  a  small  subset  of  major  pro¬ 
duct  categories — a  few  hundred  of  over  33,000  products  AFOS  is  re¬ 
quired  to  handle.  Another  major  limitation  was  that  the  test  did 
not  provide  results  over  a  range  of  system  load  conditions. 

("Load"  refers  to  the  volume  of  messages  in  the  system,  which 
normally  reflects  the  number  of  users.)  AFOS  is  very  load  sensi¬ 
tive,  and  its  performance  can  degrade  under  a  heavy  workload.  The 
AFOS  test  was  generally  performed  under  a  1 ight-to-moderate  work¬ 
load.  As  a  result,  the  test  did  not  measure  the  potential  perform¬ 
ance  of  the  system  in  full  operation.  The  test  also  excluded  an 
appropriate  standard  for  system  failures.  The  number  of  system 
failures  occurring  per  day  exceeded  the  original  standard. 

Another  limitation  of  the  test  was  that  it  did  not  include 
full  tests  of  "live"  information  as  specified  in  two  critical  NWS 
requirements.  According  to  AFOS  specifications,  the  system  must 
(1)  deliver  products  from  the  SMCC  to  field  offices,  (2)  transmit 
local  meteorological  observations  to  the  SMCC,  and  (3)  transmit 
this  data  from  the  SMCC  to  the  FAA .  During  the  test  NWS  attempted 
only  limited  transmission  of  live  SMCC  data  to  demonstrate  SMCC's 
ability  to  send  data  to  FAA.  As  a  result,  NWS  has  insufficient 
information  to  determine  whether  AFOS  can  adequately  perform 
the  required  function  of  transmitting  data  to  FAA.  Demonstrat¬ 
ing  full  performance  of  this  feature  should  be  critical  to  the 
final  decision  to  either  implement  AFOS  nationwide  or  discon¬ 
tinue  its  use,  but  NWS  chose  not  to  emphasize  this  capability  in 
the  test. 

Moreover,  insufficient  information  was  collected  during  the 
test  to  enable  projections  of  the  costs  and  benefits  of  implement¬ 
ing  AFOS  at  200  weather  stations  nationwide.  Because  of  the  vast 
difference  between  the  objectives  of  AFOS  and  the  system's  actual 
performance,  cost-benefit  information  on  its  current  operation  is 
critical.  In  particular,  NWS  did  not  collect  information  on  a 
major  cost  item,  the  personnel  support  required  by  AFOS. 


In  addition,  the  statistical  evaluation  method  used  for  the 
AFOS  test  was  deficient.  ADP  system  tests  are  normally  conducted 
at  three  levels:  best  case,  worst  case,  and  most  probable.  The 
AFOS'  test  plan  called  for  conducting  most  of  the  test  under  the 
best  or  most  probable  case  conditions.  As  a  result,  the  test  did 
not  demonstrate  whether  the  system  could  adequately  perform  under 
a  worst  case  condition,  thereby  identifying  needed  corrections 
to  system  components.  Also,  the  test  did  not  demonstrate  that  the 
system  can  perform  satisfactorily  under  a  worst  case  method  if  im¬ 
plemented  in  200  weather  stations  under  fully  loaded  conditions. 
AFOS  should  adequately  perform  under  a  worst  case  assumption  be¬ 
fore  it  is  considered  for  nationwide  implementation. 

NEED  FOR  A  MORE  COMPREHENSIVE 
PLAN  FOR  MEETING  ADP  NEEDS 

NWS  continues  to  pursue  two  goals:  implementing  AFOS  and 
concurrently  developing  a  new  system.  NWS  has  not  yet  developed 
a  detailed  plan  to  carry  out  these  objectives.  Further,  the 
NWS  approach  has  not  provided  needed  information,  including  a 
comprehensive  strategy  in  support  of  such  plans.  For  example, 

NWS'  plans  do  not  include  a  schedule  to  (1)  further  validate 
and  test  AFOS'  performance,  particularly  for  untested  segments, 

(2)  conduct  a  cost-benefit  analysis  of  AFOS  prior  to  full  imple¬ 
mentation,  and  (3)  resolve  the  system  design  constraints. 

A  full  test  for  AFOS  should  include  measurement  data  to 
determine  the  effect  on  AFOS'  performance  of  design  deficiencies 
that  are  inherent  in  its  hardware,  software,  and  telecommunica¬ 
tions  components.  By  not  focusing  on  the  design  issues,  NWS  is 
not  in  a  position  to  estimate  the  effects  of  these  deficiencies 
and/or  evaluate  the  cost  and  reasonableness  of  correcting  them. 

A  full  test  should  also  provide  information  to  determine  the  ex¬ 
tent  to  which  the  software,  hardware,  and  telecommunications  are 
deficient  and  to  what  extent  they  are  correctable.  In  addition, 
the  test  should  address  the  overlapping  and  discrete  functions 
to  be  performed  by  two  systems  in  parallel — AFOS  and  the  current 
communications  system.  Because  both  systems  perform  the  same 
basic  function  of  providing  information  to  field  offices  and  both 
systems  will  be  operating  in  parallel  for  at  least  3  years,  AFOS 
should  be  tested  in  terms  of  this  environment.  The  assessment 
should  take  into  account  other  information,  such  as  benefits  and 
associated  costs,  that  would  be  needed  to  conduct  a  cost-benefit 
analysis . 

To  adequately  address  the  range  of  constraints,  limitations, 
and  problems  involved,  NWS  should  develop  a  comprehensive  ADP 
plan  that  includes: 

1.  Both  long-range  and  short-range  approaches  to  data 

processing  accompanied  by  strategies  to  carry  out  these 
plans . 
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2.  An  economic  analysis  that  defines  goals  and  objectives, 
formulates  appropriate  assumptions,  identifies  alterna¬ 
tives,  and  determines  costs  and  benefits  of  identified 
alternatives . 

3.  An  updated  requirements  analysis  to  reflect  information 
needs  that  are  consistent  with  agency  mission  and  pro¬ 
gram  objectives. 

4.  The  impact  of  an  advanced  system  development  program  on, 
and  its  relationship  to,  other  agency  systems  and  other 
relevant  programs  that  involve  satellite  and  radar 
imagery  and  local  applications. 

5.  The  application  of  new  information  technology  that  would 
overcome  the  deficiencies  of  the  current  AFOS  design. 

6.  An  assessment  of  the  level  of  management  involvement 
needed  to  effectively  oversee  and  direct  the  various 
system  development  projects. 

7.  The  appropriate  organizational  structure,  including  that 
of  the  project  management  office,  needed  for  future 
systems  projects. 

8.  The  number  and  type  of  personnel  required  for  three 
systems — the  current  system,  AFOS,  and  the  new  system 
to  be  developed. 

9.  Results  of  various  tests  describing  both  positive  and 
negative  results  with  a  description  of  corrective  action 
required . 

We  believe  that  an  NWS  assessment  of  this  information  is 
a  fundamental  prerequisite  to  a  decision  whether  to  embark  on 
national  implementation  of  AFOS,  as  well  as  to  begin  a  long- 
range  effort  to  replace  it. 

CONCLUSIONS 

The  AFOS  system  has  inherent  design  problems  which  preclude 
it  from  adequately  meeting  NWS'  needs.  Because  these  design  con¬ 
straints  are  serious,  AFOS  is  not,  and  is  unlikely  to  be,  a  cost- 
effective  or  efficient  system. 

During  the  past  several  months  NWS  has  made  a  concerted  ef¬ 
fort  to  correct  some  deficiencies  and  demonstrated  during  the 
August-September  1981  validation  test  that  AFOS'  performance  is 
now  improved.  Based  on  our  observations  at  various  test  sites 
and  other  test  information,  we  concur  that  AFOS  has  been  improved. 
These  recent  improvements  indicate  that  AFOS  has  at  least  the 
capabilities  of  the  current  communications  system  operated  for  NWS 
by  FAA . 


2.  An  economic  analysis  that  defines  goals  and  objectives, 
formulates  appropriate  assumptions,  identifies  alterna¬ 
tives,  and  determines  costs  and  benefits  of  identified 
alternatives . 

3.  An  updated  requirements  analysis  to  reflect  information 
needs  that  are  consistent  with  agency  mission  and  pro¬ 
gram  objectives. 

4.  The  impact  of  an  advanced  system  development  program  on, 
and  its  relationship  to,  other  agency  systems  and  other 
relevant  programs  that  involve  satellite  and  radar 
imagery  and  local  applications. 

5.  The  application  of  new  information  technology  that  would 
overcome  the  deficiencies  of  the  current  AFOS  design. 

6.  An  assessment  of  the  level  of  management  involvement 
needed  to  effectively  oversee  and  direct  the  various 
system  development  projects. 

7.  The  appropriate  organizational  structure,  including  that 
of  the  project  management  office,  needed  for  future 
systems  projects. 

8.  The  number  and  type  of  personnel  required  for  three 
systems — the  current  system,  AFOS,  and  the  new  system 
to  be  developed. 

9.  Results  of  various  tests  describing  both  positive  and 
negative  results  with  a  description  of  corrective  action 
required . 

We  believe  that  an  NWS  assessment  of  this  information  is 
a  fundamental  prerequisite  to  a  decision  whether  to  embark  on 
national  implementation  of  AFOS,  as  well  as  to  begin  a  long- 
range  effort  to  replace  it. 

CONCLUSIONS 

The  AFOS  system  has  inherent  design  problems  which  preclude 
it  from  adequately  meeting  NWS1  needs.  Because  these  design  con¬ 
straints  are  serious,  AFOS  is  not,  and  is  unlikely  to  be,  a  cost- 
effective  or  efficient  system. 

During  the  past  several  months  NWS  has  made  a  concerted  ef¬ 
fort  to  correct  some  deficiencies  and  demonstrated  during  the 
August-September  1981  validation  test  that  AFOS'  performance  is 
now  improved.  Based  on  our  observations  at  various  test  sites 
and  other  test  information,  we  concur  that  AFOS  has  been  improved. 
These  recent  improvements  indicate  that  AFOS  has  at  least  the 
capabilities  of  the  current  communications  system  operated  for  NWS 
by  FAA. 
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In  view  of  this  demonstrated  improvement  in  performance  and 
the  expenditure  of  about  $100  million  for  the  project,  we  conclude 
that  NWS  should  be  permitted  to  operate  AFOS  until  1989,  but  only 
if  NWS  completes  a  full  economic  assessment  showing  that  interim 
implementation  of  AFOS  for  an  8-10  year  period  is  cost  effective. 
We  emphasize  that  the  recent  AFOS  validation  test  does  not  provide 
sufficient  information  for  a  final  determination  to  implement  AFOS 
nationally.  Without  further  testing,  analysis,  and  planning,  NWS 
cannot  determine  the  most  effective  course  of  action. 

RECOMMENDATIONS 


We  recommend  that  the  Secretary  of  Comirerce  direct  NWS  to: 

— Conduct  a  detailed  cost-benefit  analysis  before  deciding 
on  full  implementation. 

— Conduct  a  test  of  the  untested  segments  of  AFOS  before 
deciding  to  implement  it  nationwide. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

NWS*  response  to  our  report  was  based  on  a  draft  version  of 
this  report,  dated  June  29,  1981.  In  that  draft  we  suggested 
that  NWS  should  immediately  abandon  the  AFOS  system.  NWS  dis¬ 
agreed  with  our  position  and  believes  that  the  existing  AFOS  sys¬ 
tem  should  be  implemented  nationally  during  fiscal  year  1982  as 
the  primary  weather  information  system.  As  noted  above,  we  no 
longer  believe  that  AFOS  should  be  abandoned.  In  this  report  we 
are  recommending  that  national  implementation  be  conditioned  on 
a  full  assessment  of  AFOS  and  its  alternatives. 

In  commenting  on  our  June  29,  1981,  draft  report,  NWS  stated 
that  AFOS  meets  or  exceeds  the  original  system  requirements  as 
specified  in  the  June  1976  program  development  plan.  NWS'  re¬ 
ference  to  the  1976  document  as  the  original  requirement  is  mis¬ 
leading.  The  1976  document  referred  to  by  NWS  is  a  general  over¬ 
view  description  of  AFOS.  This  document  was  prepared  2  years  after 
the  original  specifications  and  is  not  a  program  specifications 
or  requirements  document.  NWS  developed  a  detailed  set  of  program 
requirements  in  1974  which  became  the  basic  design  document  for 
AFOS.  AFOS  still  does  not  meet  the  NWS-developed  specifications 
as  cited  in  its  original  1974  system  requirements  document.  In 
fact,  following  its  1981  technical  assessment,  NWS  itself  noted 
that  AFOS  could  not  meet  its  original  program  specifications. 

As  a  total  project,  AFOS  was  designed  to  accomplish  two 
broad  objectives:  (1)  automate  the  functions  performed  by  the 
current  system  and  (2)  provide  new  capabilities  to  allow  NWS  to 
take  advantage  of  improvements  in  meteorology.  With  the  recent 
improvements  made  by  NWS,  it  appears  that  AFOS  will  be  able  to 
meet  the  first  objective — automating  the  current  system.  Fur¬ 
ther,  AFOS  has  greater  capability  and  capacity  than  the  current 
communications  system  in  terms  of  processing  speeds,  information 
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storage,  and  the  ability  to  use  computer  programs  and  process 
data  in  field  offices.  However,  AFOS  cannot  provide  the  new 
capabilities  it  was  intended  to  deliver  to  its  users.  There¬ 
fore,  delivery  of  the  additional  capabilities  NWS  requires  will 
be  delayed  until  a  new  system  is  developed  to  replace  AFOS. 

In  response  to  our  draft  report  and  prior  to  the  August- 
September  1981  validation  test,  NWS  stated  that  AFOS  had  been 
tested  and  was  fully  capable  of  meeting  "operational  needs." 
However,  in  a  January  1981  test  of  AFOS,  the  program  test  man¬ 
ager  concluded  that  AFOS  was  unable  to  meet  operational  require¬ 
ments.  Based  on  the  test,  he  stated  that,  "Because  of  the  prob¬ 
lems  associated  with  using  AFOS,  most  of  the  test  monitors, 
operators  and  site  managers  felt  that  a  severe  weather  incident 
would  cause  a  total  collapse  of  the  AFOS  operation."  AFOS  has 
not  yet  satisfactorily  passed  a  complete  system  performance 
test  under  full  load  conditions,  and  until  it  shows  it  can  do 
so,  it  is  not  feasible  to  state  that  AFOS  is  fully  capable  of 
meeting  operational  needs. 

NWS  system  specifications  state  that  before  AFOS  can 
become  an  operational  system  it  must  accomplish  four  essential 
tasks:  (1)  deliver  data  to  the  field  offices,  (2)  permit  the 

manipulation  of  data  by  the  forecaster,  (3)  transmit  information 
from  the  field  office  to  the  SMCC,  and  (4)  deliver  the  field  data 
from  the  SMCC  to  the  FAA  system.  Currently  NWS  is  using  AFOS  to 
accomplish  only  the  first  two  tasks. 

NWS  stated  that  AFOS'  backup  system  is  perfectly  acceptable, 
thoroughly  tested,  and  meets  all  NWS  requirements.  The  original 
design  of  AFOS  had  five  system  backups,  and  according  to  NWS 
specifications,  all  five  are  needed  to  meet  the  minimal  require¬ 
ments  of  an  operational  system.  Of  the  five  AFOS  backup  systems, 
two  work  poorly  according  to  actual  test  results,  two  have  been 
removed,  and  the  remaining  system  has  not  been  adequately  tested. 
Therefore,  the  backup  systems  provide  little  assurance  that  AFOS 
can  effectively  meet  its  requirement  for  a  reliable  and  stable 
system. 

In  its  attempt  to  improve  AFOS'  reliability,  NWS  has  con¬ 
ducted  numerous  tests  of  the  field  offices'  ability  to  remain 
operational  with  only  one  computer  instead  of  two.  The  assess¬ 
ment  disclosed  that  normal  operations  at  a  field  office  cannot  be 
supported  by  a  single  computer.  Further,  an  independent  contrac¬ 
tor  concluded  that  because  of  the  system's  design,  one  computer 
cannot  satisfactorily  maintain  a  site's  system.  We  agree  with 
this  assessment. 

NWS  stated  that  it  has  corrected  all  the  problems  with  AFOS 
and  that  the  system  now  meets  or  exceeds  all  performance  require¬ 
ments.  However,  many  of  our  observed  problems  with  AFOS  are  in¬ 
herent  in  its  basic  design  and  cannot  be  corrected  in  a  period  of 
a  few  months,  if  ever,  short  of  replacing  the  entire  system. 
Further,  NWS  did  not  attempt  to  correct  any  of  the  basic  design 


problems.  As  a  result,  NWS  has  not  resolved  and,  as  stated  in  4  ~ 
response,  does  not  intend  to  resolve  any  of  the  design  problems 
identified  by  us  and  by  NWS  technical  staff.  NWS  plans  to  resolve 
the  system's  basic  design  problems  by  developing  a  new  system, 
which  could  be  completed  by  the  late  1980's. 

NWS  plans  to  leave  the  current  system  in  place  as  a  backup 
to  AFOS  until  1984  to  minimize  service  interruptions  whenever 
AFOS  fails  to  function.  By  this  action  AFOS  will  have  been  in 
parallel  operation  for  about  5  years.  This  length  of  time  to 
operate  two  systems  in  parallel  is  not  in  accordance  with  accepted 
management  practices  and  strongly  suggests  that  NWS  still  believes 
it  needs  more  time  to  make  essential  improvements  in  the  AFOS  sys¬ 
tem.  In  Federal  agencies  and  private  industry  a  commonly  accepted 
objective  is  to  have  systems  in  parallel  operation  for  the  shortest 
possible  time.  This  approach  is  used  because  parallel  operations 
are  usually  expensive,  place  a  significant  burden  on  operating 
staffs,  disrupt  operations,  and  delay  effective  use  of  the  new 
system. 

The  NWS  decision  to  continue  parallel  operations  for  an 
additional  3  years  is  indicative  of  AFOS'  problems  and  supports 
our  position  that  AFOS  is  not  likely  to  become  a  fully  operational 
system.  We  further  contend  that  NWS  will  probably  retain  the  cur¬ 
rent  FAA  system  as  backup  until  NWS  develops  a  new  system  (sched¬ 
uled  for  completion  in  1989)  or  until  FAA  drops  the  current 
system,  whichever  occurs  first. 

Further,  the  overrun  of  5  years  for  developing  AFOS  has 
extended  the  development  period  to  10  years.  This  10-year 
development  phase  is  twice  the  length  of  time  the  system  will  be 
in  operation  before  it  is  completely  replaced  by  a  new  system.  A 
2  to  1  ratio  of  a  development  phase  to  an  operational  time  period 
is  excessive  and  highly  unusual.  This  analysis  of  years  expended 
on  development  is  based  on  the  starting  date  of  the  development 
phase,  which  began  in  1974.  It  is  expected  to  end  in  1984  when 
NWS  plans  to  replace  the  current  communications  system.  The  5- 
year  figure  for  operations  is  that  period  between  1984  and  the 
scheduled  1989  completion  of  the  new  system  design  and  develop¬ 
ment  phase. 

NWS'  comments  are  summarized  in  appendix  I  and  presented  in 
their  entirety  in  appendix  II. 
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AGENCY  RESPONSES  AND  OUR  EVALUATION 

After  reviewing  our  draft  report,  NWS  strongly  disagreed  with 
a  substantial  portion  of  it.  (NWS'  response  is  presented  in  its 
entirety  as  appendix  II.)  After  analyzing  its  comments,  we  deter¬ 
mined  that  NWS  disagrees  on  16  key  issues. 

This  appendix  focuses  on  these  16  issues  and  presents  NWS' 
comments  and  our  evaluations.  We  have  also  summarized  NWS'  com¬ 
ments  and  our  evaluation  in  the  digest  and  at  the  end  of  each 
chapter,  as  appropriate. 

In  reviewing  NWS'  comments  to  our  report,  we  offer  the  follow¬ 
ing  observations: 

— Much  of  the  material  presented  by  NWS  is  primarily  factual 
in  nature.  Generally,  we  have  not  taken  exception  to  this 
information.  However,  we  did  not  confirm  its  accuracy. 

— NWS  frequently  stated  in  broad  terms  that  information  in 
our  draft  is  either  erroneous  or  trivial,  but  it  provided 
no  specific  information  or  evidence  to  support  its  posi¬ 
tions.  We  are  therefore  not  in  a  position  to  respond  to 
these  points. 

— NWS  frequently  summarized  its  overall  concerns  and  dis¬ 
agreements  with  our  report. 

After  observing  NWS'  August-September  1981  test  of  AFOS  and 
based  on  preliminary  test  results,  we  modified  our  overall  recom¬ 
mendation  that  NWS  discontinue  all  use  and  development  of  AFOS. 

This  required  modifying  our  recommendations  in  chapters  4,  5, 

6,  and  7,  which  we  have  done.  In  essence,  our  modified  recommen¬ 
dations  acknowledge  the  recent  improvements  in  performance  and 
therefore  provide  a  qualified  "yes"  for  moving  to  the  next  step 
in  the  implementation  of  AFOS,  which  we  suggest  is  to  reappraise 
the  new  evidence  of  improvements  in  AFOS'  performance. 

In  our  opinion,  the  NWS  validation  test  of  AFOS  in  August- 
September  1981,  which  was  limited  to  selected  products  and  opera¬ 
tions,  does  not  justify  full  implementation  of  AFOS  on  a  nationwide 
basis.  We  continue  to  have  serious  reservations  in  terms  of  cost 
effectiveness  and  efficiency  if  AFOS  were  implemented  nationwide 
in  over  200  weather  stations.  Where  appropriate,  we  have  made 
changes  in  the  final  report  to  reflect  changed  conditions,  im¬ 
provements,  and  our  modified  positions.  Further,  we  point  out 
that  AFOS'  basic  design  problems  still  exist  and  that  NWS  has  not 
initiated  appropriate  actions  to  resolve  them. 

1.  AGENCY  RESPONSE 

"No  major  problems  in  AFOS  software,  hardware,  or 

telecommunications  remain  that  aftect  its  essential 

functional  capabilities." 
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OUR  EVALUATION 


We  acknowledge  improvement  in  AFOS'  performance.  Tt  is 
operating  and  carrying  out  limited  functions. 

However,  deficiencies  we  have  identified  in  those  components 
are  inherent  in  their  basic  designs.  For  example,  the  software 
component,  which  includes  three  major  segments — the  operating  sys¬ 
tem,  the  applications  software,  and  the  telecommunications 
software — has  severe  limitations,  in  particular,  the  software 
segments  are  so  tightly  integrated  in  the  computer's  memory  that 
major  software  malfunctions  can  be  expected  to  occur.  That  is, 
the  organizational  structure  of  the  software  does  not  consist  of 
independent  modules  that  would  permit  changes  to  one  module  with¬ 
out  affecting  other  modules.  As  the  workload  resulting  from  a 
fully  implemented  system  significantly  increases,  the  potential 
for  software  malfunctions  likewise  increases.  The  inappropriate 
structure  of  integrated  software  makes  the  entire  software  vulner¬ 
able.  In  particular,  as  one  software  segment  fails,  the  remaining 
software  segments  will  most  likely  fail.  For  example,  the  dead¬ 
lock  problem  with  the  software  component  per sits.  When  the  com¬ 
puter  attempts  to  process  two  or  more  tasks  which  compete  with 
the  same  resource,  a  system  malfunction  occurs  and  the  system  is 
incapable  of  processing  other  tasks. 

Also,  AFOS  as  currently  designed  is  not  capable  of  inter¬ 
facing  with  or  incorporating  other  systems  under  development. 

For  example,  AFOS  lacks  the  capability  to  either  transmit  radar 
and  satellite  imagery  data  to  field  offices  or  receive  or 
process  information  from  remote  area  meteorological  observa¬ 
tion  stations.  Consequently,  AFOS  meets  limited  system  require¬ 
ments,  but  it  cannot  meet  all  the  major  needs  specified  in  its 
original  requirements. 

2.  AGENCY  RESPONSE 

"AFOS  is  sufficiently  stable  and  reliable  to  use 
as  the  primary  system;  the  existing  AFOS  system 
is  now  serving  as  the  primary  system  in  two  of 
the  four  contiguous  NWS  regions;  and  operational 
adequacy  will  be  demonstrated  during  August  and 
September  1981." 

OUR  EVALUATION 

Through  recent  onsite  observations  and  based  on  preliminary 
test  results  from  the  August-September  1981  test,  we  acknowledge 
an  improvement  in  reliability.  We  note  that  NWS  has  (1)  removed 
a  number  of  system  features  to  reduce  complexity  and  minimize 
system  failures  and  (2)  declared  a  moratorium  on  changes  to  the 
software  to  help  stabilize  the  entire  system. 


However,  AFOS  does  not  qualify  as  a  primary  system  until  it 
actually  replaces  the  current  communications  system  and  performs 
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its  essential  functions.  To  be  the  primary  system,  AFOS  should 
operate  as  such  for  an  acceptable  period  of  time  in  parallel  with 
the  current  system.  But  it  plans  to  operate  AFOS  in  parallel 
for  nearly  3  more  years,  rather  than  several  months.  Not  only  is 
this  lengthy  parallel  operation  inconsistent  with  acceptable  man¬ 
agement  practices,  but  it  has  not  been  justified  on  a  cost- 
benefit  basis.  NWS  has  yet  to  demonstrate  that  AFOS  can  meet 
prescribed  requirements  and  acceptable  standards  and  practices  in 
such  areas  as  flexibility,  expandability,  reliability,  backup 
capability,  maintainability,  and  operability. 

3.  AGENCY  RESPONSE 


"The  most  important  backup  capabilities  designed  into 
AFOS  have  been  implemented,  and  the  pre-AFOS  communi¬ 
cations  arrangements  will  be  retained  as  an  added 
backup  through  FY  1984." 

OUR  EVALUATION 


Although  certain  backup  capabilities  have  been  tested  and 
the  procedures  appear  to  work  satisfactorily,  limitations  and 
deficiencies  continue  to  exist.  For  example,  when  the  System 
Monitoring  and  Coordination  Center's  computer/telecommunications 
system  fails,  its  backup  procedures  do  not  assure  continuous 
operation  with  minimum  interruptions.  Continuous  operations  are 
highly  dependent  on  the  SMCC,  the  heart  of  the  AFOS  telecommuni¬ 
cations  network.  All  weather  service  information  must  be  trans¬ 
mitted  through  the  SMCC  from  the  large  computer  system  at  the 
National  Meteorological  Center  to  the  200  field  offices  and  from 
field  offices  to  the  NMC .  Also,  information  flowing  to  and  from 
the  NMC  from  all  sources  must  pass  through  the  SMCC,  including 
information  exchanged  with  other  weather  services  (FAA,  Air  Force, 
and  Navy) . 

Further,  the  operation  of  computer  systems  dispersed 
nationwide  is  completely  dependent  upon  the  nationwide  telecom¬ 
munications  network  for  all  essential  functions.  If  any  of  the 
WSFO  computer  systems  fail  to  function  for  10  minutes  or  longer, 
an  operator  must  manually  pull  a  bypass  switch  to  cut  the  failed 
system  out  of  the  telecommunications  loop.  Failure  to  do  so  will 
result  in  lost  data  and  operational  problems  in  the  adjoining 
stations  on  the  network. 

In  addition,  particular  limitations  and  deficiencies  with 
the  backup  procedures  exist  in  WSFOs  where  the  system  has  two  mini¬ 
computers.  When  one  computer  fails,  the  second  computer  must  con¬ 
tinue  to  perform  its  normal  telecommunications  function  as  well 
as  handle  the  other  computer's  load.  In  this  instance,  the  second 
computer  operating  in  a  degraded  mode  results  in  slower  processing 
of  fewer  functions.  This  condition  is  compounded  by  the  computer's 
lack  of  core  memory  to  handle  the  increased  workload,  thus  result¬ 
ing  in  a  high-risk  situation. 
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4.  AGENCY  RESPONSE 

"Improvements  in  AFOS  have  been  incorporated  success¬ 
fully  during  the  last  year,  and  NWS  believes  that 
some  investment  in  selective  improvements  (not 
fundamental  redesign  and  development) ,  retaining  the 
existing  hardware  and  most  of  the  software,  will 
further  improve  system  performance  and  facilitate 
future  software  maintenance  and  enhancement." 

OUR  EVALUATION 


Based  on  our  observation  of  NWS'  August-September  1981  AFOS 
validation  test,  we  agree  that  NWS  has  made  improvements  in  AFOS. 
However,  the  improvements,  in  part,  resulted  from  removing  exist¬ 
ing  capabilities  to  relieve  a  shortage  of  core  memory  to  improve 
reliability.  The  improvements  have  also  resulted  from  the  exten¬ 
sive  use  of  personnel  and  management  attention  to  increase  system 
discipline  and  apply  standard  operating  procedures.  Nonetheless, 
we  are  concerned  that  this  level  of  support  cannot  be  sustained 
over  the  next  8  to  9  years. 

NWS  stated  that  the  current  plan  is  to  freeze  the  system  "as 
is"  and  hold  all  changes  to  a  minimum.  AFOS  technical  and  man¬ 
agement  personnel  have  determined,  and  we  agree,  that  AFOS'  prob¬ 
lems  stem  from  fundamental  design  flaws  which  can  be  resolved  only 
by  a  major  redevelopment.  In  short,  the  potential  for  substantial 
improvement  in  AFOS  has  been  precluded  by  NWS'  approaches  to  solv¬ 
ing  immediate  problems.  By  stating  that  it  plans  to  freeze  the 
system,  NWS  clearly  states  that  the  AFOS  design  will  not  be 
improved . 

5.  AGENCY  RESPONSE 

"The  cost  of  operating  and  maintaining  AFOS  is  close 
to  the  projections  made  in  1976  when  adjusted  for  the 
effects  of  inflation  on  salaries  and  spare  parts." 

OUR  EVALUATION 


Since  we  did  not  review  this  aspect,  we  have  no  basis  to 
accept  or  reject  NWS'  statement.  Due  to  the  lack  of  cost  data 
available  from  the  NWS  accounting  system,  we  did  not  attempt  to 
determine  the  cost  to  maintain  and  operate  the  system.  In  es¬ 
sence,  our  review  focused  on  the  cost  to  develop  AFOS.  NWS  it¬ 
self  provided  the  cost  figure  of  $13  million  per  year  for  opera¬ 
tions  and  maintenance  that  we  have  used.  The  only  adjustment  we 
made  was  to  allocate  overhead  to  the  $13  million  as  required  by 
NOAA  and  NWS  accounting  procedures. 

6 .  AGENCY  RESPONSE 


"The  existing  AFOS  system  meets  all  significant 
original  requirements  except  for  early  withdrawal 
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of  teletypewriter  and  facsimile  capabilities, 
retained  as  backup,  and  substantially  exceeds 
original  expectations  in  some  areas  such  as 
forecaster  assistance." 

OUR  EVALUATION 


NWS  stated  that  AFOS  meets  or  exceeds  the  original  require¬ 
ments  as  specified  in  the  June  1976  Program  Development  Plan. 

NWS'  reference  to  the  1976  document  as  the  original  requirement 
is  misleading.  The  1976  document  is  a  general  overview  descrip¬ 
tion  of  AFOS.  This  document  was  prepared  2  years  after  the  origi¬ 
nal  specifications  and  does  not  meet  the  standards  for  a  program 
specifications  or  requirements  document.  In  1974  NWS  developed 
a  detailed  set  of  program  requirements  that  became  the  basic 
design  document  for  AFOS.  We  still  contend  that  AFOS  does  not 
meet  NWS  specifications  as  cited  in  its  original  1974  system 
requirements  document.  In  fact,  following  its  1981  technical 
assessment,  NWS  itself  concluded  that  AFOS  is  unable  to  meet  the 
original  program  specifications. 

7.  AGENCY  RESPONSE 

"The  initial  AFOS  system  was  never  intended  or  designed 
to  meet  all  future  requirements ,  and  particularly  not 
those  cited  by  GAO." 

OUR  EVALUATION 

NWS'  original  program  specifications  document  of  1974  and  all 
subsequent  specifications  documents  have  specifically  stated  that 
the  use  of  radar  and  satellite  data,  as  well  as  the  new  remote 
area  meteorological  observing  systems  under  development,  would  be 
part  of  AFOS.  Further,  these  requirements  documents  place  a  top 
priority  on  flexibility  and  the  ability  of  AFOS  to  handle  new 
systems  and  program  changes.  In  addition,  a  top  priority  was  as¬ 
signed  to  the  requirement  that  the  meteorologists,  inexperienced 
in  ADP  and  telecommunications,  would  be  able  to  operate  and  use 
AFOS  without  intensive  training.  As  NWS  itself  concluded  in  its 
March  1981  technical  assessment,  AFOS  is  unable  to  meet  these 
requirements . 

AFOS  is  also  unable  to  meet  NWS'  original  specifications  on 
frequency  of  software  breakdowns,  time  to  repair  hardware  and  cor¬ 
rect  software,  backup  levels  of  protection,  and  a  number  of  com¬ 
puterized  routines  to  support  the  forecasters.  Our  position  is 
also  supported  by  NWS'  assessment  that  AFOS  is  not  capable  of 
handling  radar  and  satellite  imagery  data  and  other  requirements 
in  its  original  specifications.  Further  support  for  our  position 
is  illustrated  by  NWS'  decision  to  begin  developing  a  new  system 
to  meet  mission  needs  and  to  overcome  AFOS1  limitations  and  defi¬ 
ciencies.  A  new  system  <-o  replace  AFOS  by  1989  was  one  item  which 
did  not  appear  in  NWS*  requirements  documents. 
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8.  AGENCY  RESPONSE 

"AFOS  is  extremely  valuable  to  NWS  operations;  it 
significantly  speeds  up  warnings  and  forecasts  as 
planned,  saves  large  numbers  of  staff-hours  annually 
through  automation  of  routine  tasks,  liberates  time 
and  resources  for  service  improvements,  and  estab¬ 
lishes  a  mode  of  operation  amenable  to  future 
improvements . " 

OUR  EVALUATION 

In  spite  of  its  limitations,  AFOS  does,  we  agree,  speed  up 
warnings  and  forecasts.  We  disagree  with  NWS  that  AFOS  has  re¬ 
sulted  in  personnel  savings.  As  we  noted  in  the  report,  NWS  has 
added  119  positions  to  develop  and  operate  AFOS;  it  has  no  basis 
to  attribute  personnel  reductions  to  AFOS.  NWS  continues  to  dis¬ 
count  the  significant  personnel  costs  for  training,  managing  a 
telecommunication  system,  and  providing  the  management  and  techni¬ 
cal  support  required  by  a  complex  ADP  system.  Most  of  these 
services  were  previously  supplied  by  FAA  at  no  cost  to  NWS. 

Until  NWS  begins  collecting  and  accounting  for  personnel 
costs  as  required  by  NOAA  and  NWS  accounting  procedures,  NWS  will 
continue  to  underestimate  AFOS  personnel  costs.  A  key  reason  for 
establishing  these  accounting  procedures  was  to  provide  manage¬ 
ment  with  the  true  cost  of  ADP  systems  development  and  operations. 
Without  this  information  management  has  no  basis  on  which  to 
determine  total  personnel  costs  to  its  organization  by  specific 
projects.  NWS  continues  to  neglect  the  collection  and  analysis 
of  cost  information  for  AFOS. 

AGENCY  RESPONSE 

9.  "GAO's  estimate  that  $116  million  could  be  saved  by  not 
operating  AFOS  for  8  years  does  not  consider  important 
costs  of  trying  to  turn  back  to  operations  without  AFOS, 
and  totally  ignores  the  major  capabilities  and  efficien¬ 
cies  that  would  be  lost." 

OUR  EVALUATION 

As  disclosed  in  our  report,  the  cost  savings  originally 
identified  by  NWS  as  AFOS'  benefits  will  not  be  realized.  During 
our  review  of  AFOS  we  repeatedly  requested  that  NWS  prepare  a 
cost-benefit  analysis.  Yet,  NWS  repeatedly  stated  that  it  was  un¬ 
necessary  at  this  point  in  time  and  that  it  had  no  plans  to  conduct 
such  an  analysis.  Without  such  benefit  and  cost  information, 
neither  NWS  nor  any  cognizant  organization,  including  GAO,  NOAA, 
or  congressional  committees,  is  in  a  position  to  compare  the  bene¬ 
fits  against  costs. 
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10.  AGENCY  RESPONSE 


"Reverting  to  pre-AFOS  systems  involves  much  more  than 
retaining  connection  to  FAA's  teletypewriter  circuits, 
including  refurbishing  or  replacing  aging  equipment 
of  various  types,  substantially  increasing  field  per¬ 
sonnel,  and  making  other  potentially  expensive 
ad j  ustments . " 

OUR  EVALUATION 

The  current  communications  system,  without  the  AFOS  system, 
has  been  operating  for  many  years  and  continues  to  meet  NWS'  basic 
requirements  in  all  six  regions. 

Further,  NWS  stated  that  it  plans  to  retain  the  current 
system  until  1984.  Although  additional  funds  would  be  required 
to  maintain  or  replace  the  current  communications  system,  con¬ 
tinuing  to  operate  AFOS  would  be  considerably  more  expensive. 

For  example,  in  its  March  1981  technical  assessment,  NWS  iden¬ 
tified  $12  to  $15  million  as  the  amount  needed  to  upgrade  AFOS. 
This  does  not  include  expenditures  to  resolve  basic  AFOS  design 
deficiencies  and  constraints. 

11 .  AGENCY  RESPONSE 

"Even  current  services  could  not  be  maintained  using 
pre-AFOS  systems  without  a  major  investment  and 
upheaval  in  NWS,  if  at  all.  The  pre-AFOS  systems  do 
not  meet  even  1974  NWS  requirements;  that  is  why  the 
AFOS  program  was  initiated  to  provide  many  capabili¬ 
ties  not  included  at  all  in  the  previous  systems." 

OUR  EVALUATION 


Current  services  can  be  maintained  without  AFOS  as  evidenced 
by  the  fact  that  NWS  is  providing  all  current  services  using  its 
current  communications  system,  not  AFOS.  In  addition,  NWS  stated 
that  it  plans  to  retain  the  current  communications  system  as  a 
backup  to  AFOS.  This  backup  plan,  therefore,  supports  our  posi¬ 
tion  that  as  AFOS  fails  to  perform  and  does  not  provide  needed 
services,  the  current  communications  system  can  provide  these 
services . 

It  is  also  important  to  note  that  with  the  exception  of 
limited  data  processing  capability  for  local  programing,  AFOS 
does  not  have  any  major  capabilities  not  contained  in  the  current 
system.  AFOS  basically  performs  the  same  major  functions  as  the 
current  system.  The  additional  capabilities  planned  for  AFOS, 
such  as  handling  radar,  satellite,  and  remote  observations,  have 
been  deleted  from  the  system.  NWS  expects  to  include  these  re¬ 
quirements  in  the  new  system  it  plans  to  develop. 
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12.  AGENCY  RESPONSE 


"Totally  different  kinds  of  personnel  are  involved  in 
operating  AFOS  and  in  developing  a  new  system;  NWS  can 
accomplish  both  concurrently." 

OUR  EVALUATION 


NWS  does  not  adequately  account  for  the  agency's  experience 
with  AFOS  during  the  very  lengthy  and  costly  development  phase 
which  began  in  1974  and  is  continuing.  Field  operating  personnel 
invested  a  significant  amount  of  time  in  developing,  testing,  and 
operating  AFOS.  This  time  expenditure  is  likely  to  be  repeated 
when  NWS  is  again  involved  in  a  new  system  development  effort 
and  concurrently  maintaining  AFOS  to  meet  its  needs. 

Also,  NWS  does  not  adequately  address  the  agency's  severe 
lack  of  management  and  technical  personnel  who  are  experienced  in 
ADP  development.  Developing  a  new  system,  in  addition  to  sup¬ 
porting  AFOS  and  maintaining  the  current  system,  will  place  an 
excessive  burden  on  both  management  and  technical  personnel. 
Further,  by  not  adequately  assessing  the  total  personnel  needs 
based  on  a  system  to  properly  account  for  personnel  costs,  NWS 
will  continue  to  pass  over  the  very  real  and  heavy  investment  of 
its  personnel  in  AFOS  development  and  operations. 

13.  AGENCY  RESPONSE 


"The  cost  of  a  new  system  meeting  all  present  and 
future  requirements  recommended  by  GAO  is  unknown,  and 
GAO ' s  estimate  of  $125  to  $150  million  is  not  based  on 
any  substantive  analysis  or  knowledge  of  the  require¬ 
ments;  and 

"the  assumption  that  more  than  $100  million,  now 
planned  for  expenditure  in  small  pieces  over  eight 
years,  can  be  gathered  up  for  a  procurement  is  at  best 
doubtful . " 

OUR  EVALUATION 

NWS  objected  to  our  estimate  that  the  development  cost  of  a 
new  system  would  be  $125  to  $150  million.  We  believe  that  this 
estimate  is  reasonable  in  light  of  AFOS  development,  which  when 
completed  will  have  cost  about  $150  million,  with  the  effects  of 
inflation  on  personnel  costs  for  software  development  (the  major 
cost  of  an  ADP  system) ,  it  is  very  unlikely  that  a  new  system 
would  cost  less  than  $125  to  $150  million. 

NWS  also  disagreed  with  our  position  that  over  $100  million 
scheduled  to  be  spent  in  small  pieces  over  8  years  could  be  ac¬ 
cumulated  for  the  procurement  of  a  new  system.  We  agree  with  NWS 
and  would  like  to  point  out  that  we  neither  stated  nor  assumed 
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this  was  the  case.  It  is  our  position  that  a  new  system  develop¬ 
ment  project  should  be  approved  and  funded  by  the  Congress  and 
not  internally  reprogramed  from  operations  and  maintenance  funds. 

14.  AGENCY  RESPONSE 

"The  development  of  large,  complex  systems  that  break 
frontiers  should  be  expected  to  encounter  some  prob¬ 
lems;  NOAA's  experience  shows  that  proceeding  in  a 
series  of  ambitious  but  prudent  steps  results  in  fewer 
problems  than  making  a  single  giant  leap  to  a  new  sys¬ 
tem  incorporating  even  more  novel  features  than  AFOS, 
as  proposed  by  GAO." 

OUR  EVALUATION 

We  acknowledge  the  new  frontiers  penetrated  by  NWS  in 
initiating  the  AFOS  project.  We  also  acknowledge  NWS'  position 
that  it  intends  to  replace  AFOS  with  a  new  system.  This  new  sys¬ 
tem  is  designed  to  meet  requirements  identified  by  NWS  in  1974 
and  to  incorporate  additional  requirements  to  meet  mission  needs. 
Our  report  does  not  recommend  specific  capabilities  for  AFOS, 
but  it  identifies  the  areas  in  which  AFOS  lacks  capabilities 
prescribed  for  it  in  the  1974  requirements.  Further,  our  reser¬ 
vations  are  directed  toward  the  cost  and  advisability  of  operat¬ 
ing  AFOS. 

15.  AGENCY  RESPONSE 


"The  schedule  to  complete  AFOS  slipped  2  years,  not  5 
years. " 

OUR  EVALUATION 

AFOS  will  be  fully  operational  only  when  the  current  system 
is  removed  in  1984  and  all  stations  are  using  AFOS  as  the  primary 
system.  In  developing  AFOS  and  in  its  internal  plans  prepared 
prior  to  responding  to  our  report,  the  development  phase  was  to 
end  in  1979  and  the  removal  date  of  the  current  system  was  the 
termination  date  of  the  development  phase.  NWS'  revised  position 
is  that  the  development  phase  will  end  in  1982  when  it  expects 
all  scheduled  offices  to  be  using  AFOS,  even  though  the  current 
system  will  remain  in  operation  until  1984.  Further,  in  develop¬ 
ment  projects  in  the  private  sector  and  in  Government,  the  date 
the  current  system  is  removed  normally  indicates  the  end  of  a 
development  phase.  By  this  measure  AFOS  is  5  years  behind 
schedule . 

16.  AGENCY  RESPONSE 


"NWS  used  management  practices  appropriate  to  the 
stage  of  development  and  the  organizational  realities 
of  NWS." 
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OUR  EVALUATION 

As  we  noted  in  our  report,  technical  assessments  by  independent 
contractors  as  well  as  by  us  have  noted  major  management  problems 
in  AFOS  development.  NWS  has  not  followed  standard  Government 
guidance  and  acceptable  management  practices  for  system  management, 
accounting,  and  software  development.  Such  management  deficiencies 
have  significantly  contributed  to  AFOS'  delays  and  problems. 

In  March  1981,  following  a  detailed  analysis  of  AFOS,  the 
project  manager  in  his  final  report  summarized  NWS'  management 
problems  as  follows: 

"The  consultants  we  retained  in  recent  years,  as  well 
as  the  GAO  team  currently  reviewing  the  AFOS  program, 
have  stressed  that  other  organizations  successfully 
undertaking  automation  programs  of  the  scope  and 
complexity  of  AFOS  were  forced  to  make  significant 
organizational  and  management  changes.  These  were 
necessary  to  accommodate  this  level  of  automation 
and  to  institute  much  higher  levels  of  classical 
systems  engineering  and  management  disciplines  than 
the  NWS  has  invoked  to  date.  We  have  been  repeatedly 
warned  that  we  risk  continued  problems  and  failure 
unless  we  exercise  a  higher  degree  of  discipline 
in  the  establishment  of  requirements,  development, 
operation,  change  management,  documentation,  support 
and  program  management  than  is  exercised  in  the 
customary  practices  of  the  NWS. 

"Our  experience  during  the  past  year  seems  to  confirm 
these  views.  Many  of  our  problems  can  be  traced  to 
failures  of  management  and  systems  liscipline.  These 
have  lead  to  unrealistic  plans,  schedules  and  expec¬ 
tations.  Priorities  in  allocation  of  staff  and 
resources  have  been  obscure,  confusing  and  incon¬ 
sistent  through  the  organization.  The  flow  of  up-to- 
date,  authoritative,  honest  information  has  been 
spotty  and  slow,  and  the  exercise  of  management 
direction  often  has  been  weak  and  fuzzy.  Collegial 
decision-making  has  promoted  participation  at  the 
expense  of  focus,  clarity  and  decisiveness." 

We  concur  with  this  assessment.  It  should  also  be  noted  that 
NWS  attempted  to  develop  one  of  the  largest  distributed  data  base 
systems  ever  designed,  in-house,  without  trained  ADP  personnel, 
without  increasing  staffing  levels,  and  without  modifying  its 
organizational  structure.  Given  these  constraints,  management  and 
technical  problems  are  not  surprising.  ADP  development  projects 
require  strong  central  management  and  heavy  emphasis  on  systems 
discipline. 


75 


APPENDIX  I 


APPENDIX 

sibility  f or ^ADp6r) ^  NWS  must  retain  f  n 

contractors  to  Dcf i°3Qcts •  We  believe  h 11  mana9ement  resoon 

Personnel  on  o'S7?*?n  technical taikt'^wever '  tha t  by  Sl?o 

beet  atlVL'  address?"1  a"d  "Penological 


76 


APPENDIX  II 


APPENDIX  II 


/  v  \ 

T  ,  , 


\  1  . 


T  J? 


UNITED  STATES  DEPARTMENT  OF  COMMERCE 
Office  of  Inspector  General 

W.ishifHjtnn  [)<!  IM ]?“$() 


AUG  3  I 


Mr.  Henry  Eschwege 
Director,  Community  and  Economic 
Development  Division 
U.  S.  General  Accounting  Office 
Washington  ,  D.  C.  20548 


Dear  Mr.  Eschwege: 


This  is  in  reply  to  your  letter  of  June  29,  1981,  requesting 
comments  on  the  draft  report  entitled  “The  National  Weather 
Service  Should  Abandon  the  Automation  of  Field  Operations  and 
Services  System . " 

We  have  reviewed  the  enclosed  comments  of  the  Deputy 
Admi ni st rator ,  National  Oceanic  and  Atmospheric  Administration 
for  the  Department  of  Commerce  and  believe  they  are  responsive  to 
the  matters  discussed  in  the  report. 


Sincerely, 


Enel osure 
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Comments  of 

National  Oceanic  and  Atmospheric  Administration  * 

Department  of  Commerce 

on 

GAO  Draft  of  a  Proposed  Report 

on 

"The  National  Weather  Service  Should  Abandon 
the  Automation  of  Field  Operations  and  Services  Systems" 


Dated 

July  24,  1981 

* 


With  OMB  changes  of  August  26,  1981 
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OVERVIEW 


AFOS  Status  Update 

The  technical  problems  that  delayed  operational  implemen¬ 
tation  of  the  Automation  of  Field  Operations  and  Services  (AFOS) 
program  now  have  been  isolated  and  resolved.  The  performance  of 
AFOS  hardware,  software,  and  telecommunications  meets  all  func¬ 
tional  and  technical  requirements  established  by  the  National 
Weather  Service  (NWS)  as  essential  for  proceeding  with  the  formal 
operational  demonstration.  Already  two  of  the  four  contiguous 
NWS  regions  are  relying  on  AFOS  for  nearly  all  planned  service 
operations.  The  system  has  been  installed  in  all  of  the  more 
than  200  field  offices  as  planned,  the  communications  links  have 
been  established,  and  AFOS  is  supporting  some  critical  opera¬ 
tional  needs  at  all  of  these  stations.  The  NWS  regions  using 
AFOS  as  the  primary  system  report  that  it  significantly  speeds 
the  preparation  and  dissemination  of  severe  weather  and  flash 
flood  warnings,  one  of  the  major  original  goals  of  the  program. 
Equally  important,  AFOS  provides  staff  relief  to  NWS  field 
offices  by  automating  numerous  time  consuming  manual  tasks  and 
simplifying  many  others. 

A  formal  operational  demonstration  has  been  designed  to 
verify  that  full  weather  service  operations  conducted  using  AFOS 
as  the  primary  weather  information  system  meet  rigorous,  quan¬ 
titative  standards  of  reliability  and  timeliness.  The  hardware, 
software,  network  maintenance  and  logistics  systems,  and  other 
support  elements  already  are  in  their  final  configuration  for  the 
demonstration;  the  field  personnel  of  two  entire  NWS  regions  are 
fully  trained  and  proficient;  and  the  demonstration  will  be 
completed  by  the  end  of  September  1981  as  scheduled.  Assuming 
successful  completion  of  this  operational  demonstration  plus 
prior  completion  of  an  extensive  technical  validation  of  the  AFOS 
system  covering  the  last  18  months  and  a  comprehensive  program  of 
field  training  and  verification  of  operational  and  support 
procedures,  NWS  plans  to  proceed  with  national  implementation  of 
the  system.  The  NWS  plans  that  its  regions  will  be  formally 
commissioned,  one-by-one  in  a  phased  process,  resulting  in  all 
four  regions  operational  by  the  end  of  FY  1982. 

Basic  Issues 


Two  factors  stand  out  in  the  basic  conflict  between  the  pre¬ 
ceding  summary  of  the  status  and  plans  for  AFOS  implementation 
and  the  major  conclusions  and  recommendations  of  the  GAO  draft 
report.  First,  there  has  been  positive  progress  in  the 
development,  testing,  and  initial  field  use  of  the  AFOS  system 
since  the  GAO  completed  its  visitation  and  data  collection 
activities  several  months  ago.  Second,  NOAA  and  GAO  have  signif¬ 
icantly  different  perspectives  on  the  nature  of  weather  service 
operations  and  the  requirements,  relative  importance,  and  value 
of  various  technical  and  service  capabilities  associated  with 
AFOS. 
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In  essence,  there  are  two  fundamental  issues  that  separate 
NOAA  and  GAO. 

1.  NOAA  contends  that  the  AFOS  system  will  meet  the  agency's 
requirements  for  support  of  weather  service  operations  in 
essential  accordance  with  its  design  and  original  program 
goals,  and  plans  to  implement  the  system  operationally 
beginning  this  fall  with  phased  extension  across  the  country 
by  the  end  of  FY  1982.  In  contrast,  GAO  concluded  that  the 
system  cannot  meet  its  original  requirements  and  must  be 
completely  redeveloped. 

2.  NOAA  believes  that  AFOS  should  serve  as  the  primary  infor¬ 
mation  system  supporting  NWS  field  operations  for  the  next 
7-9  years,  and  plans  to  develop  a  new  system  incorporating 
major  added  capabilities  for  implementation  near  the  end  of 
the  decade.  In  contrast,  GAO  recommends  that  NWS  abandon 
further  efforts  to  develop  and  use  the  existing  AFOS  system 
and  concentrate  its  resources  on  the  development  of  a  new 
system  to  replace  AFOS.  In  the  interim  period,  GAO  proposes 
that  NWS  should  revert  to  total  reliance  on  the  systems  and 
procedures  in  use  prior  to  the  introduction  of  AFOS. 

Performance  vs.  Requirements 

The  first  issue  outlined  above  involves  two  essential 
elements:  a)  the  actual  performance  capabilities  of  the  system, 

and  b)  the  requirements  of  the  agency.  The  performance  of  the 
system  is  observable,  and  in  principle,  subject  to  agreement  as 
"fact. "  Unfortunately,  GAO  observed  the  system  at  an  earlier 
stage  of  development  when  a  number  of  technical  problems  had  not 
yet  been  resolved.  In  the  last  few  months  there  has  been  marked 
improvement  and  stablization  in  system  performance.  In  fact,  two 
of  the  four  contiguous  NWS  regions  currently  are  using  AFOS  as 
the  primary  weather  information  system  supporting  normal,  day-to- 
day  service  operations.  Nearly  all  of  the  approximately  200  NWS 
offices  equipped  with  AFOS  are  depending  on  the  system  for  some 
important  functions  such  as  preparation  of  forecast  and  warning 
messages.  The  test  and  demons trat ion  activities  of  the  next 
2  months  will  provide  quantitative  observations  of  the  perform¬ 
ance  of  the  system  under  real,  operational  conditions,  24  hours 
per  day,  over  an  extended  period. 

The  question  of  requirements  is  more  subtle.  It  could 
reduce  down  into  a  legalistic  argument,  in  a  historical  context, 
with  reference  to  various  documents  associated  with  the  program 
and  system  development  process.  This  is  a  natural  route  for 
auditing.  Inevitably,  questions  of  interpretation,  intent, 
changes,  and  the  adequacy  of  documentation  arise  from  this 
approach.  The  NWS  believes  that  the  AFOS  system  does  in  fact 
meet  the  requirements  set  forth  when  the  program  was  initiated  in 
1974.  A  more  fundamental  argument,  however,  is  that  normally  the 
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user  agency  is  in  the  best  position  to  judge  whether  a  system 
meets  its  requirements  or  not,  whether  they  have  changed  or  not, 
and  whether  they  are  adequately  documented  or  not.  In  subsequent 
sections,  specific  capabilities  already  in  use  and  their  rele¬ 
vance  to  weather  service  operations  are  summarized. 

We  recognize  that  system  performance  and  associated  require¬ 
ments  comprise  much  more  than  functional  capabilities.  System 
reliability,  maintainability,  backup  capability,  flexibility, 
complexity  of  operations,  cost  of  operations,  and  similar  factors 
are  important.  After  careful  review  and  evaluation  NWS  has 
judged  the  AFOS  system  design  and  performance  satisfactory  in  all 
of  these  aspects.  Justifications  supporting  this  assessment  are 
presented  in  subsequent  sections.  NWS  judgments  of  the  tech¬ 
nical  performance  and  operational  suitability  of  the  system  are 
based  on  actual  current  experience  with  the  system.  In  contrast, 
GAO's  conclusions  are  based  on  observations  only  during  earlier 
development  stages  when  problems  are  normal  and  expected.  GAO's 
major  conclusions  rest  heavily  on  extrapolations  based  on  general 
principles  and  experience  with  other  ADP  systems. 

This  is  not  to  deny  that  there  have  been  significant 
problems  with  the  system  in  the  past,  nor  to  suggest  that  there 
are  no  system  characteristics  that  could  be  improved. 
Nevertheless,  the  existing  AFOS  system  as  designed,  developed, 
installed,  and  used  in  the  field,  meets  essent’’  requirements  of 
service  operations.  One  notable  deviation  from  our  original  plan 
is  the  decision  to  retain  teletypewriter  and  facsimile  capabili¬ 
ties  at  AFOS-equipped  offices  for  the  first  2-3  years  of 
operation. 

The  rationale  for  retaining  these  systems  is  threefold. 
First,  they  provide  a  proven  backup  in  case  of  unexpected 
failures  in  the  AFOS  system.  This  backup  approach  was  an 
integral  part  of  the  decision  to  streamline  the  design  of  the 
System  Monitoring  and  Coordination  Center  (SMCC)  near 
Washington,  D.C. ,  thereby  limiting  some  built-in  AFOS  backup 
capabilities.  Details  are  presented  later,  but  NWS  considers 
this  backup  arrangement  to  obviate  the  concerns  expressed  by  GAO 
about  the  deferral  of  some  previously  planned  backup  modes  within 
the  AFOS  system.  Second,  other  weather  field  offices  not 
equipped  with  AFOS  must  rely  on  these  systems  anyway  until  they 
are  equipped  with  leased,  remote  terminals  during  the  next  2-3 
years.  Finally,  it  is  prudent  to  retain,  the  old  familiar  system 
for  an  interim  period  as  a  "safety  net"  during  the  transition  to 
an  entirely  new  technology  and  mode  of  operation. 

Value  of  AFOS 


The  second  major  issue  identified  above  rests  more  on  the 
different  value  systems  in  which  the  judgments  are  made.  The 
contrasting  conclusions  result  from  different  judgments  not  only 
on  what  is  needed  by  NWS  and  when,  but  also  on  what  is  possible, 
what  is  practical,  what  is  acceptable,  and  what  is  "worth  it." 
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NOAA  and  GAO  agree  that  eventually  (we  project  possibly  the 
end  of  the  decade)  NWS  can  and  should  develop  a  new  system  which 
incorporates  additional  capabilities  that  cannot  be  added  to  AFOS 
without  major  design  changes.  The  original  Program  Development 
Plan  for  AFOS  recognized  that  an  additional  development  phase 
would  be  needed ,  although  it  was  somewhat  optimistic  (in 
hindsight)  in  projecting  the  mid-1980's  for  this  stage.  Much  of 
the  research  and  development  work  needed  for  an  advanced  system 
is  underway  in  NOAA,  research  institutions,  and  industry.  A  spe¬ 
cific  planning  effort  directed  toward  acquiring  such  a  system  has 
been  initiated  by  NWS.  NOAA  recognizes  that  major  and  time- 
consuming  program  and  budget  decision  processes  in  both  the  exe¬ 
cutive  and  legislative  branches  will  be  required  before  such  a 
program  could  proceed  beyond  a  modest  planning  effort. 

However,  today  and  every  day  from  now  until  a  new  system  is 
available,  the  Nation  requires  weather  services.  NOAA  must  pro¬ 
vide  these  services  in  the  most  effective  and  efficient  way 
available.  We  are  convinced  that  AFOS  provides  a  significantly 
more  effective  and  efficient  way  of  providing  weather  services 
than  has  ever  been  available.  After  coming  this  far  with  AFOS, 
it  is  even  difficult  to  imagine  how  to  conduct  our  operations 
without  AFOS.  The  termination  of  AFOS,  for  whatever  reason, 
would  be  a  major  set  back  to  progress  in  weather  services. 

Obviously,  if  the  AFOS  system  failed  to  function  in  a  way 
that  is  essential  for  service  operations,  it  not  only  shouldn't 
out  couldn't  be  used  as  the  primary  operational  system.  We 
believe  that  we  have  shown  that  the  system  will  demonstrably  meet 
essential  requirements.  In  this  case,  a  question  still  remains, 
is  it  "worth  it."  To  answer  this,  we  must  answer  what  are  the 
alternatives  and  what  are  their  comparative  advantages  and 
d isadvantages. 

The  draft  GAO  report  sketches  out  one  rationale  for  such  a 
comparison,  based  on  the  concept  of  "savings  from  not  using  AFOS 
between  now  and  the  end  of  the  decade."  It  is  asserted  that 
$116M  would  be  saved,  a  number  surely  large  enough  to  be 
impressive.  However,  it  is  based  mainly  on  multiplying  GAO's 
estimated  annual  "operating  cost"  by  a  large  number  of  years,  and 
thus  could  be  made  even  larger  by  deferring  a  new  system  even 
longer!  Quite  apart  from  disagreements  on  the  precise  amount  of 
the  cost  estimates  for  operating  AFOS,  this  approach  is  inade¬ 
quate  in  other  respects.  It  assumes  that  the  only  other  "cost" 
that  should  be  considered  when  providing  weather  services  without 
AFOS  is  the  current  NWS  cost  of  using  the  FAA  teletypewriter 
system  multiplied  by  the  number  of  years  considered.  Even  this 
cost  factor  is  misleading  because  FAA  pays  the  bulk  of  the  cost 
for  the  current  system  and  plans  to  discontinue  it  in  favor  of  a 
system  using  AFOS-type  technology  during  the  projected  time 
interval. 
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Even  more  serious  are  the  implied  assumptions  in  such  a 
rationale,  e.g.,  that  operating  with  AFOS  and  with  the  FAA  tele¬ 
typewriter  network  are  somehow  equivalent  (they  aren't),  or  that 
it  is  even  possible  to  continue  operations  with  the  current  tech¬ 
nology  at  any  cost  for  this  time  period;  or  that  the  relief  from 
manual  tasks  that  automation  provides  is  without  value,  or 
conversely,  that  NWS  could  maintain  essential  services  without 
additional  costs.  Perhaps  most  crucially,  maintaining  services 
without  AFOS  would  require  additional  personnel,  and  they  have 
become  even  more  scarce,  valuable,  and  costly  than  ever  during 
recent  years. 

Compounding  these  deficiencies  in  GAO's  analysis  is  the 
complete  neglect  of  the  comparative  qual i ty  of  services  using  the 
two  systems.  We  have  demonstrated  that  AFOS  significantly  speeds 
the  preparation  and  dissemination  of  weather  warnings  as  compared 
to  the  current  system  (one  of  the  major  goals  of  AFOS).  This  is 
an  important  value  of  AFOS,  and  its  loss  should  be  an  important 
cost  in  an  economic  analysis.  Furthermore,  AFOS  increases  the 
quality  of  analysis  and  forecast  products  in  numerous  ways  by 
providing  new  tools  to  assist  forecasters  in  analysis  and 
interpretation  of  weather  information.  The  improvements  in 
service  quality  will  grow  with  continuing  use  of  the  system  and 
exploitation  of  its  inherently  new  and  powerful  capabilities  to 
process  and  display  information. 

It  is  clear  that  it  is  a  gross  and  sterile  over¬ 
simplification  to  equate  not  using  AFOS  to  a  cost  savings  of 
5116M  —  or  any  numerical  figure  quotable  solely  in  dollars. 

Outline  of  NQAA's  Detailed  Comments 

The  remainder  of  NOAA's  comments  following  this  overview  is 
organized  in  accordance  with  the  chapters  of  the  draft  GAO  report 
with  the  exception  of  chapter  1.  GAO's  chapter  1  deals  mainly 
with  background  and  organization  of  the  National  Weather  Service 
without  conclusions  or  recommendations.  There  are  a  few  factual 
errors  and  NWS  is  working  directly  with  GAO  to  correct  these. 

At  the  beginning  of  chapter  2,  we  have  pulled  together  in 
one  place  an  up-to-date  summary  of  the  current  status  of  the  AFOS 
system.  This  is  especially  important  because  GAO  gathered  nearly 
all  of  the  information  used  to  prepare  the  draft  report  during 
the  period  between  July  1980  and  March  1981.  A  number  of  the 
criticisms  refer  to  situations  or  practices  which  have  sub¬ 
sequently  been  altered.  In  particular,  most  of  GAO's  obser¬ 
vations  concerning  the  actual  performance  of  the  system,  and 
judgments  derived  from  those  observations  date  from  a  major 
engineering  system  test  conducted  under  field  operating  con¬ 
ditions  during  January  1981.  Since  that  time,  engineering 
testing  has  been  completed,  and  problems  that  GAO  staff  con¬ 
sidered  intractable  have  been  overcome.  The  system  performance 
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now  meets  all  NWS  primary  functional  and  technical  requirements, 
and  we  are  beginning  to  rely  on  AFOS  as  the  primary  weather 
information  system  in  a  substantial  number  of  NWS  field  offices. 
AFOS  telecommunications,  data  organization  and  manipulation,  ana¬ 
lysis  and  forecast  preparation,  automatic  dissemination  network 
control  and  backup  capabilities  all  now  meet  essential  NWS 
requirements.  The  current  levels  of  these  capabilities  are 
outlined  at  the  beginning  of  chapter  2. 

Chapter  2  also  includes  a  section  on  costs  and  schedules 
which  should  correct  several  misconceptions  that  seem  to  exist  on 
these  topics.  A  confusion  between  early  budget  estimates 
referring  to  obligational  authority  (exclusive  of  overhead  and 
routine  cross-utilization  of  existing  support  functions)  and 
accountable  costs  is  clarified.  This  permits  a  more  reasonable 
comparison  of  actual  versus  projected  costs.  Similarly,  the 
actual  evolution  of  the  AFOS  schedule,  some  of  it  in  reponse  to 
changes  in  budget  profile,  is  presented.  The  rationale  that 
leads  to  GAO's  surprising  conclusion  that  AFOS  is  5  years  behind 
schedule  is  examined,  and  NOAA  concludes  that  2  years  is  a  more 
reasonable  estimate.  Finally,  chapter  2  of  these  comments 
addresses  the  value  that  NOAA  already  derives  from  AFOS  and 
expects  to  achieve  from  future  use  of  AFOS,  and  compares  these  to 
early  expectations  when  the  program  was  launched. 

Chapter  3  addresses  the  management  of  the  AFOS  program.  A 
summary  of  the  current  management  structure,  policies,  and  prac¬ 
tices  is  provided  along  with  the  rationale  for  them.  Again,  some 
significant  adjustments  have  been  made  which  may  account  for  part 
of  the  large  disagreement  on  management  between  NOAA  and  GAO,  but 
we  suspect  the  explanation  rests  more  directly  on  the  fact  that 
NWS  cannot  limit  its  management  considerations  so  narrowly  to  ADP 
issues  as  GAO  recommends.  We  conclude  that  the  successful  devel¬ 
opment  of  AFOS  to  its  present  level,  ready  for  final  operational 
demonstration,  is  a  better  measure  of  program  management  than  the 
alleged  departure  from  general  principles  of  ADP  management. 

Chapter  4  summarizes  the  nature  and  status  of  AFOS  software. 
The  facts  presented  refute  the  GAO  claims  that  AFOS  software  is 
inadequate  and  explains  how  sound  technical  and  management  tech¬ 
niques  were  used  to  correct  software  problems  encountered  earlier 
in  the  program.  NOAA  considers  that  successful  development  of 
reliable  AFOS  software  that  accomplishes  such  a  wide  array  of 
concurrent  functions  is  a  significant  achievement.  The  great 
complexity  and  tight  integration  of  the  software  is  acknowledged 
by  NWS,  along  with  the  consequence  that  further  changes  and 
improvements  are  difficult  and  time  consuming.  However,  these 
limitations  do  not  affect  the  ability  of  the  system  to  support 
weather  service  operations  as  planned.  Furthermore,  after  the 
initial  system  has  been  commissioned,  NWS  plans  to  relieve  these 
limitations  and  do  so  without  additional  resources  beyond  the 
level  requested  for  FY  1982. 
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Chapter  5  summarizes  the  nature  and  status  of  AFOS 
telecommunications.  Of  all  the  technical  issues  related  to  AFOS, 
this  one  has  provoked  comment  and  concerns  of  the  most  diverse 
character  and  origin.  Perhaps  this  occurs  because  the  super¬ 
ficial  concepts  of  network  structure  can  be  understood  by  anyone, 
and  most  persons  agree  that  network  concepts  exist  that  appear 
superior  in  one  way  or  another  to  the  loop  design  selected  for 
AFOS.  NOAA  bases  its  program  decision  on  the  fact  that  the 
selected  design  successfully  meets  all  operational  requirements 
for  the  system  and  does  so  with  reasonable  resources.  NOAA  will 
continue  to  review  alternative  possibilities  and  be  alert  for 
opportunities  to  improve  performance  or  achieve  economies,  but 
tnere  is  no  known  basis  for  rejecting  the  existing  successful 
design.  It  is  anticipated  that  major  extensions  of  weather 
information  systems  currently  under  initial  study  for  the  end  of 
the  decade  will  likely  require  a  more  advanced  design. 

Chapter  6  summarizes  the  nature  and  status  of  AFOS  hardware. 
The  facts  clearly  contradict  the  assessment  of  GAO.  In  addition 
to  the  timing  problem  of  GAO's  observations  mentioned  before, 
there  are  other  possible  explanations  for  this  discrepancy. 

First,  in  the  complex  AFOS  system,  it  is  never  obvious  to  an 
observer  whether  a  functional  failure  at  a  site  is  due  to  a  soft¬ 
ware  or  a  hardware  problem  or  even  to  a  problem  within  the 
telephone  companies'  line  and  switching  systems.  After  comple¬ 
tion  of  validation  testing,  NWS  has  determined  that  nearly  all  of 
the  "failures"  observed  during  the  period  of  GAO's  visitation 
were  caused  by  software  "bugs"  which  were  readily  corrected. 
Second,  the  GAO  report  points  out  ♦'hat  the  equipment  used  in  AFOS 
does  not  take  advantage  of  the  most  recent  advances  in  electronic 
design.  This  deficiency  exists  in  every  major  system  procured  by 
the  government  and  would  exist  in  the  new  system  that  GAO  propo¬ 
ses  be  developed  to  replace  AFOS,  because  advances  in  electronics 
occur  rapidly  while  system  procurement,  integration,  testing  and 
introduction  to  service  take  several  years.  Again,  the  proper 
test  is  performance,  and  the  AFOS  hardware  meets  all  specifica¬ 
tions  for  reliability  and  maintainability.  The  criticism  that 
the  cost  of  operations  and  repair  will  be  excessive  is  unsubstan¬ 
tiated  and  counter  to  NWS  experience  with  AFOS  to  date  and  with 
analogous  systems. 

Chapter  7  examines  the  validity  of  the  composite  assessment 
of  AFOS.  The  basic  tenets  that  are  used  to  support  GAO's  conclu¬ 
sion  that  AFOS  should  be  abandoned  are  considered  and  found  to  be 
invalid  in  the  other  chapters  as  noted  above.  The  GAO  report 
concludes  that  the  AFOS  system  cannot  work.  In  fact,  the  AFOS 
system  already  does  work  as  shown  not  only  by  engineering  tests 
but  by  actual  reliance  on  the  system  by  many  NWS  offices  to  pro¬ 
vide  primary  operational  support  (even  though  the  system  has  not 
yet  been  formally  commissioned).  Any  remaining  doubts  about  the 
adequacy  of  the  system  to  support  operations  should  be  settled  by 
the  objective,  formal  operational  demonstration  scheduled  to  be 
completed  by  the  end  of  September  1981. 
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The  GAO  also  questions  whether  the  introduction  of  AFOS  is 
justified.  This  is  not  a  new  question  and  was  considered  by 
NOAA,  the  Executive  Branch,  and  Congress  before  the  program  was 
authorized.  The  current  system  as  developed  achieves  the  objec¬ 
tives  anticipated  except  for  the  withdrawal  of  existing  teletype¬ 
writer  and  facsimile  systems  which  would  save  an  estimated  $2.6M 
annually.  NWS  plans  to  achieve  the  latter  savings  by  the  end  of 
FY  1984.  Nearly  all  of  the  investment  of  resources  (7  years  of 
effort  and  nearly  $100M  according  to  GAO)  already  has  been  made 
to  achieve  these  objectives. 

The  only  argument  remaining  might  be  that  it  costs  too  much 
to  operate  the  system  (about  $13M  annually  by  NOAA's  estimate  or 
$15M  according  to  GAO  including  unrecoverable  overhead  costs). 

GAO  compares  this  only  to  their  estimate  of  $3. 5M  annually  for 
NOAA  to  connect  to  FAA's  teletypewriter  circuits.  Significant 
additional  costs  would  be  required  under  this  option  in  order  to 
refurbish  and  replace  equipment  that  would  be  essential  with  the 
"old  system"  but  is  being  phased  out  with  AFOS;  to  add  approxi¬ 
mately  150  technicians  to  operate  labor  intensive  services,  such 
as  NOAA  Weather  Wire;  to  assume  the  entire  cost  of  operating 
national  teletypewriter  circuits  following  FAA's  planned  ter¬ 
mination  of  them  in  the  mid-1980's  and  other  costs  associated 
with  termination  of  the  AFOS  personnel,  facilities,  and 
equipment.  These  direct  costs  would  be  magnified  by  the  loss  in 
productivity  of  NWS  personnel,  degradation  in  the  quality  of 
service  and  lost  opportunity  to  make  planned  improvements  in 
services.  The  NWS  allocates  about  $100M  annually  just  to  pay  the 
people  who  provide  weather  services,  and  a  comparable  amount  to 
provide  support,  systems  development,  etc.  NOAA  believes  that  an 
investment  in  AFOS  of  this  proportion,  one  that  promises  to 
modernize  and  improve  the  entire  mode  of  operations  of  NWS,  and 
one  which  provides  the  only  known  path  to  improved  services 
without  increasing  personnel  levels,  is  worth  it. 
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CHAPTER  2 

CURRENT  STATUS  AND  COST  OF  AFOS 


CURRENT  STATUS  OF  AFOS 


There  have  been  improvements  made  in  the  performance, 
capability,  and  stability  of  the  AFOS  software,  hardware,  and 
telecommunications  in  the  last  year  and  a  half.  The  system's 
capability  to  assist  the  forecaster  meets  or,  in  some  cases, 
exceeds  original  plans.  This  has  been  achieved  through  an  inno¬ 
vative  and  systematic  test  and  design  validation  program  and  a 
phased  introduction  of  AFOS  into  actual  field  operations. 

Contrary  to  the  performance  observed  by  the  GAO  team 
approximately  6  months  ago,  AFOS  is  being  used  as  the  primary 
tool  by  forecasters  in  the  preparation  and  delivery  of  weather 
services  by  two  of  the  four  contiguous  NWS  regions.  NWS  antici¬ 
pates  by  the  end  of  FY  1982  to  have  commissioned  AFOS  operations 
at  all  planned  sites  in  all  four  contiguous  regions. 

AFOS  brings  to  Weather  Service  operations  technological 
capabilities  heretofore  unavailable  to  weather  forecasters.  In 
addition  to  providing  an  advanced  telecommunications  capability 
far  exceeding  the  capability  of  current  teletypewriter  circuits 
in  use  at  field  sites,  AFOS  provides  the  capabilities  for 
sophisticated  local  data  organization  and  manipulation,  powerful 
new  analytical  capabilities,  as  well  as  significantly  enhanced 
forecast  preparation  features.  AFOS  brings  to  NWS  operations  the 
capabilities  of  automated  dissemination  of  warnings  and  services. 
These  unique  capabilities  provided  by  AFOS  directly  improve  the 
productivity  and  effectiveness  of  NWS  personnel  which  results  in 
improved  timeliness  and  quality  of  NWS  warnings  and  forecast 
serv ices. 

AFOS  Telecommunications 


The  AFOS  communications  system  is  proving  to  be  signifi¬ 
cantly  faster  than  the  existing  teletypewriter  and  facsimile 
systems.  It  is  being  used  today  to  distribute  on  a  routine, 
24-hour-a-day ,  7-day-a-week  basis,  the  products  and  data 
necessary  for  operational  use.  This  information  is  automatically 
transmitted  with  built  in  error-detection  and  retransmission 
capabilities.  The  products  used  in  forecasting  arrive  signifi¬ 
cantly  sooner  and  provide  additional  lead  time  for  field  fore¬ 
casters  to  interpret  weather  situations  and  formulate  their  state 
and  local  forecast  products.  The  Regional  Distribution  Circuits 
are  carrying  synchronous  communications  with  fully  satisfactory 
performance  and  reliability.  Both  alphanumeric  and  graphic  pro¬ 
ducts  are  available  on  time  or  earlier.  In  the  event  of  computer 
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failure  or  telephone  line  outage,  both  automatic  and  manual 
backup  procedures  are  in  place  and  proven  under  actual  field  use. 

Data  Organization  and  Manipulation 

The  AFOS  system  incorporates  the  feature  of  a  distributed 
data  base  resident  at  each  site.  This  feature  coupled  with  the 
computer  controlled  and  managed  telecommunication  system  of  AFOS 
accomplishes  the  automatic  reception,  storage,  and  updating  of 
the  station's  data  base.  Station  staff  are  being  relieved  of  the 
requirement  to  handle  the  large,  continuous  stream  of  products  on 
paper  rolls  used  by  the  teletypewriter  and  facsimile  systems. 

AFOS  automatically  alerts  forecasters  when  any  product  of  special 
interest,  such  as  warnings  and  watches  for  severe  weather  and 
flood  events,  enter  the  local  office.  Thus,  the  automated  com¬ 
munication  capability  provided  by  AFOS  is  saving  valuable  time  of 
field  staff,  preventing  oversights  and  errors,  and  permitting 
increased  emphasis  on  service  and  productivity.  The  localized 
data  base  is  tailored  to  each  station's  need  and  can  be  supple¬ 
mented  by  the  high  speed  request/reply  feature  of  AFOS  making  the 
master  data  base  of  over  30,000  products  instantly  available  to 
the  forecaster. 

Analysis  and  Forecast  Preparation 

AFOS  provides  on-site  processing  capability  for  the  first 
time  to  forecast  and  warning  offices.  A  significant  portion  of 
the  system’s  computing  power  is  available  for  manipulation  and 
formulation  of  data,  analyses  and  products  —  a  capability  that 
the  Weather  Service  has  never  had  before.  Stations  are  now  using 
AFOS  to  analyze  and  compute  local  river  levels,  analyze  detailed 
moisture  and  wind  conditions  for  severe  weather  forecasting,  plot 
surface  aviation  observations,  plot  and  analyze  local  and 
regional  temperature  fields,  plot  and  analyze  data  for  fire 
weather  advisories,  analyze  upper  air  soundings,  verify  terminal 
and  zone  forecasts,  compute  snow  pack  melt,  and  many  more  local 
applications.  The  enhanced  preformat  and  text  composition  and 
editing  features  of  AFOS,  when  compared  to  a  teletypewriter 
keyboard,  are  providing  station  forecasters  the  opportunity  to 
directly  improve  the  quality  and  timeliness  of  their  products  and 
services.  In  addition,  the  data  processing  capability  frees  sta¬ 
tion  personnel  of  virtually  all  routine  manual  plotting  and 
analysis,  resulting  in  additional  efficiencies. 

Automatic  Dissemination 


AFOS  automatically  drives  various  external  circuits,  notably 
the  NOAA  Weather  Wire  Service  (NWWS).  The  NWS  devotes  a  major 
effort  to  dissemination  of  its  products  and  services,  and  NWWS  is 
an  essential  means  of  doing  so,  particularly  for  issuing  watches 
and  warnings  of  hazardous  weather.  This  has  been  a  labor- 
intensive  program  —  tearing  teletypewriter  tapes,  preparing 
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messages  for  the  nearly  continuous  NWWS  transmissions,  generating 
paper  tapes  to  drive  teletypewriter  machines,  and  manually 
feeding  the  tapes  into  reader/transmitters.  AFOS  automatically 
searches  its  data  base  for  those  products  scheduled  for  routine 
NWWS  transmission,  properly  formats  this  information,  addresses 
the  messages,  and  automatically  transmits  the  products  at  sched¬ 
uled  times  over  the  weather  wire  with  automatic  priority  for 
watches  and  warnings.  AFOS  now  performs  these  functions  automat¬ 
ically  and  eliminates  others  completely. 

System  Monitoring  and  Coordinating  Center  (SMCC) 

The  SMCC  is  designed  to  interface  with  the  National 
Meteorological  Center  (NMC)  computer,  maintain  the  master  data 
base  of  all  weather  service  products  transmitted  over  AFOS,  and 
drive  and  service  Regional  Distribution  Circuits  (RDC's). 

Because  of  its  key  role  in  the  operations  of  AFOS,  the  SMCC 
has  a  special  system  design  which  incorporates  the  same  computers 
used  at  field  sites.  This  permits  the  shared  use  of  the  AFOS 
logistics  systems.  SMCC's  design  is  modular  with  principal  func¬ 
tions  assigned  to  separate  computer  systems.  Each  major  computer 
system  is  fully  backed  up  with  a  second  system  and  automatic 
switching  equipment  to  convert  quickly  from  one  system  to  its 
backup. 

SMCC  has  been  receiving  NMC  data  and  maintaining  the  AFOS 
master  data  base  for  several  years.  Having  successfully 
completed  the  validation  tests  at  SMCC  ahead  of  schedule,  the 
SMCC  is  now  fully  configured  and  driving  each  RDC  as  it  was 
designed  to  do,  and  is  providing  network  monitoring  and  site 
assistance  services.  The  operation  of  all  computers  is  stable 
and  reliable.  Two  network  services,  master  console  dial-in  (a 
third  level  backup)  and  site  data  base  replenish,  were  deleted 
after  the  comprehensive  program  review  earlier  this  year,  both 
having  been  judged  as  not  essential  for  the  national  implemen¬ 
tation  of  AFOS.  Their  removal  also  reduced  system  complexity 
somewhat  and  contributes  to  improved  stability. 

AFOS  Back-up  Capability 

AFOS  has  sophisticated,  multiple  levels  of  operational/ 
service,  systems,  and  telecommunication  back-up  modes. 
Telecommunication  backup  is  discussed  in  detail  in  Chapter  5. 
Since  regional  precommissioning  efforts  moved  into  full  swing, 
the  development  of  detailed  operational/service  back-up  proce¬ 
dures  began,  and  they  have  been  tested  in  actual  emergency 
situations. 

The  initial  AFOS  design  for  system  backup  is  as  follows:  a 
WSFO's  equipment  complement  includes  dual  computers  and  a  Master 
Console.  (Master  Consoles  are  similar  to  "smart  terminals"  on 
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the  market  that  contain  microprocessors  and  telephone  modem 
equipment  in  addition  to  the  keyboard  and  CRT  display. )  In  nor¬ 
mal  operation,  one  computer  is  assigned  primary  communication 
responsibility  and  the  other  is  assigned  processing 
responsibility.  Each  computer  can  perform  either  principal 
function,  and  with  somewhat  reduced  capability,  can  perform  both 
functions  simultaneously.  If  both  computers  fail  at  once  (this 
is  projected  to  occur  once  in  2.5  years),  the  Master  Console  can 
dial  into  a  remote  site  and  function  like  another  console  at  the 
remote  location. 

The  operation  protocol  at  a  site  is  as  follows:  (1)  Normal 
mode  —  dual  computer  operation,  (2)  back-up  mode  —  single  com¬ 
puter  operation  (either  computer),  and  (3)  third  level 
backup  —  Master  Console  dial-in  to  a  remote  site. 

A  WSO  has  a  single  computer  and  Master  Console.  Its  opera¬ 
tions  protocol  in  event  of  a  computer  failure  is  Master  Console 
dial-in  to  a  remote  site. 

All  modes  of  backup  have  been  tested  and  are  in  use  today. 

AFOS  permits  NWS  sites  for  the  first  time  to  quickly  and 
effectively  initiate  operation/serv ice  backup  wherein  service 
responsibilities  at  a  failed  site  (due  to  catastrophic  types  of 
failures  like  lightning  hits,  tornadoes,  telecommunication  and 
power  outages,  etc.)  are  assigned  to  another  (one  or  more)  site. 
For  example,  each  WSFO  has  primary  responsibility  to  drive  a 
number  of  NWWS  circuits  in  its  state.  Using  AFOS,  servicing  NWWS 
circuits  is  an  automated  procedure  —  the  directory  of  products 
and  timetable  for  delivery  of  each  are  contained  in  the  AFOS  com¬ 
puters  which  automatically  issue  the  prescribed  products  in 
accordance  with  those  schedules.  In  order  for  the  backup  WSO  to 
assume  responsibility  to  drive  that  state's  NWWS  circuits,  WSO 
personnel  need  only  issue  a  few  commands  to  its  AFOS  computer  and 
"hit  a  button"  —  the  computer  which  has  a  duplicate  of  the  NWWS 
transmission  schedule  then  takes  over  automatically.  Other  serv¬ 
ice  operations  are  similarly  transferable  to  back-up  stations. 

Summary  of  Current  Status 

In  summary,  there  has  been  remarkable  progress  since  the  GAO 
field  visitations  to  the  NWS.  All  AFOS  hardware  and  communica¬ 
tions  are  installed  witn  all  tour  regional  loops  being  driven 
by  the  SMCC.  The  AFOS  system  is  now  being  used  in  operations 
throughout  the  NWS  and  is  the  primary  system  for  preparation  of 
weather  forecasts  and  warnings  in  the  Central  and  Western 
regions,  the  two  of  the  four  regions  most  advanced  in  operational 
use  of  the  system.  Performance  of  the  system  is  continuing  to 
improve  daily  as  field  personnel  gain  experience  in  its  operation 
and  the  improvements  learned  from  the  recently  concluded  test  and 
evaluation  efforts  are  applied.  Its  capability  to  assist  the 
forecaster  meets  or  exceeds,  in  some  areas,  original  plans. 


92 


APPENDIX  II 


APPENDIX  II 


NWS  is  prepared  to  move  forward  to  the  Operational 
Demonstration  of  AFOS.  Assuming  a  successful  outcome,  final  com¬ 
missioning  of  the  system  for  field  operations  would  occur.  The 
detailed  test  plan  has  been  prepared  for  the  Operational 
Demonstration  and  the  arrangements  for  gathering  and  analyzing 
the  demonstration  data  are  in  place. 

AFOS  SCHEDULE  AND  COSTS 


NOAA  perceives  the  AFOS  program  to  be  2  years  behind 
schedule,  not  5  years  as  GAO  contends.  The  5-year  differential 
cited  by  GAO  is  derived  using  the  estimated  operational  date  of 
August  1979  in  the  FY  1974  AFOS  Program  Development  Plan  (PDP) 
and  the  currently  estimated  date  for  the  completion  of  the 
removal  of  existing  teletypewriter  communications  by  the  end  of 
FY  1984. 

The  1976  AFOS  PDP  reflects  the  schedule  for  implementation 
of  AFOS  based  on  the  approved  funding  level;  the  schedule  in  the 
previous  1974  PDP  noted  by  GAO  was  based  on  proposed  level  of 
funding.  According  to  the  1976  PDP,  AFOS  would  be  operational  at 
all  stations  during  the  first  quarter  of  FY  1981.  Under  the 
present  schedule,  AFOS  will  be  implemented  as  the  primary  mode  of 
operation  by  the  end  of  FY  1982,  approximately  2  years  later  than 
planned  in  the  1976  PDP. 

NOAA  considers  AFOS  to  be  operational  at  the  time  it  is  the 
primary  mode  of  operation  at  all  equipped  stations  (end  of 
FY  1982)  and  not  the  protracted  time  used  by  GAO  —  the  date  by 
which  the  standby  teletypewriter  and  facsimile  circuits  are 
removed  (end  of  FY  1984). 

The  PDP  for  AFOS  was  based  on  three  sources  of  support: 

1.  New  budget  authority,  as  represented  in  the  PDP  that  sup¬ 
ported  the  budget  request. 

2.  Development  funds  which  existed  in  the  NWS  base  budget, 
to  support  fully  dedicated  personnel. 

3.  Use  of  existing  personnel  whose  functional  respon¬ 
sibilities  related  to  similar  functional  requirements 
within  the  AFOS  program;  for  example,  training, 
facilities,  procurement,  and  operations  personnel  at  NWS 
Headquarters  and  at  the  regions,  which  is  the  standard 
practice  in  NOAA. 

GAO  uses  $77. 6M  as  a  base  reference.  This  total  is  only  the 
amount  which  appears  under  new  budget  authority  (1.  above)  and 
does  not  include  all  the  funding  submitted  to  and  approved  by  the 
Congress  under  the  AFOS  sub-activity  for  the  years  FY  76  through 
FY  80.  During  those  years  NOAA  also  included  $5.9M  reprogrammed 
base  R&D  funds  to  be  used  for  AFOS  development  (2.  above). 


93 


APPENDIX  II 


APPENDIX  II 


NWS  estimates  that  the  use  of  existing  personnel  whose  func¬ 
tional  responsibilities  related  to  AFOS  (3.  above)  during  the 
period  FY  1976  through  1980  amounted  to  $3. 5M.  Therefore,  within 
the  framework  that  the  PDP  for  AFOS  was  prepared,  NWS  expended  a 
total  of  $87. 0M  through  FY  1980.  In  addition,  development  per¬ 
sonnel  were  assigned  to  AFOS  programs  in  the  last  2  years  to 
assist  in  the  concentrated  effort  to  solve  technical  problems. 

The  salaries  of  these  development  personnel  amounted  to  approxi¬ 
mately  $2M  which  should  have  been  included  in  the  cost  of  AFOS 
but  were  not.  In  summary,  the  NWS  estimates  the  cost  of  AFOS  to 
be  $89M  through  FY  1980.  This  leaves  a  difference  of  $11M  be¬ 
tween  the  NWS  estimated  costs  identified  above  and  the  $100M 
estimate  of  GAO  that  we  believe  stems  from  the  substantially 
different  accounting  procedures  used  by  NOAA  and  GAO  to  handle 
overhead. 

Accounting  Procedures 

It  is  important  to  note  that  the  NWS,  as  a  part  of  NOAA, 
must  use  NOAA's  accounting  systems.  The  NOAA  system  is  a  cost 
based  system,  was  approved  by  GAO  prior  to  implementation,  and 
has  been  in  operation  for  a  number  of  years. 

There  is  a  basic  conceptual  difference  between  the  NWS  and 
the  GAO  team  in  the  handling  of  overhead.  Overhead  is  identified 
and  planned  within  the  NOAA  system  and  is  then  automatically 
distributed  among  all  operating  projects  on  a  direct  labor  dollar 
basis.  The  system  has  always  made  the  appropriate  distribution 
of  overhead  to  AFOS  based  upon  direct  labor  dollars  incurred  and 
identified  to  AFOS.  Overhead  is  basically  a  fixed  pool  of 
distributable  costs;  it  doesn't  vary  markedly  from  year-to-year; 
and  it  was  not  significantly  impacted  —  increased  —  by  the 
advent  of  AFOS.  Since  the  distribution  of  this  overhead  is  not 
directly  controllable  —  it  is  a  function  of  the  ratio  of  direct 
labor  worked  on  AFOS  in  any  given  year  to  total  direct  labor 
worked  in  the  NWS  —  and  further  was  not  a  part  of  the  funds 
appropriated  specifically  for  AFOS,  NWS  management  did  not 
include  it  in  the  AFOS  PDP  and  has  not  included  it  in  its  identi¬ 
fication  of  resources  available  or  expended  for  AFOS. 

In  arriving  at  AFOS  development  costs,  NWS  AFOS  financial 
management  has  been,  and  continues  to  be,  focused  on  the  manage¬ 
ment  of  identifiable  and  controllable  obligations,  relating  these 
to  funds  specifically  appropriated  or  reprogrammed  to  support  the 
development,  procurement,  and  installation  of  AFOS.  This  prac¬ 
tice  is  consistent,  and  therefore  comparable,  with  the  manner  in 
which  all  financial  data  have  been  displayed  within  the  various 
AFOS  PDP's  and  have  been  represented  to  our  various  levels  of 
review  authority  and  to  the  Congress.  The  GAO's  insistence,  at 
this  stage  of  the  program,  of  including  overhead  in  their  total 
cost  computation  and  then  comparing  it  with  the  financial  projec¬ 
tions  of  the  PDP,  creates  an  end  result  where  the  true  variance 
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between  planned  and  actual  cost  is  greatly  distorted.  In  other 
words,  the  allowance  for  overhead  the  GAO  made  in  their  estimate 
of  costs  should  also  be  added  to  the  resources  identified  in  the 
PDP  for  AFOS. 

It  should  be  noted  that  while  OMB  Circular  A-109  was  pro¬ 
mulgated  in  1976,  it  was  not  implemented  in  the  Department  of 
Commerce  until  1978.  At  that  point  the  acquisition  of  the  AFOS 
system  was  so  nearly  complete  that  it  was  not  considered  to  be 
applicable. 

VALUE  OF  AFOS 

The  AFOS  program  had  two  basic  objectives:  (1)  increase  the 
lead  time  of  severe  weather  and  flood  warnings  provided  to  the 
public;  (2)  enhance  the  quality  of  forecasts  and  services  by 
freeing  the  station  personnel  from  many  of  the  manual  routine 
tasks  now  required  and  by  providing,  for  the  first  time,  the 
forecaster  with  a  local  application  processing  capability.  In 
contrast  to  the  opinion  of  GAO,  the  present  AFOS  system  performs 
all  of  these  functions  at  least  as  well  as  originally  planned. 

Warnings 

In  a  weather  station,  peak  workloads  arise  for  several 
reasons;  certain  times  of  the  day  when  the  forecasts  are  being 
updated  on  a  scheduled  basis  (this  schedule  must  adhere  to  the 
needs  of  users,  not  to  the  desires  of  the  Weather  Service);  when 
the  weather  is  changing  even  in  modest  ways;  and  during  severe 
weather  and  flooding  situations.  During  each  of  these 
situations,  the  number  of  repetitive  operations  that  require 
manual  non-professional  activity  increases  many  fold  and  in 
addition,  the  forecaster  needs  the  maximum  information  of  the 
weather  conditions  in  the  local  area,  including  special  analyses. 
The  value  of  saving  the  time  of  field  personnel  is  most  critical 
during  these  situations  and  in  fact,  it  is  precisely  during  these 
times  that  AFOS  provides  maximum  assistance. 

The  success  or  failure  of  the  Weather  Service  in  providing 
severe  weather,  especially  tornadoes,  and  flash  flood  warnings, 
depends  on  the  lead  time  that  is  given  to  the  public.  For  tor¬ 
nado  warnings,  the  average  lead  time  has  been  about  3  minutes. 

In  the  pre-AFOS  period,  the  functions  which  are  all  done  manually 
preceding  the  actual  dissemination  of  warning  included: 

1.  Recognition  that  a  potentially  hazardous  condition  did  or 
would  exist. 

2.  Determination  of  the  degree  and  extent  of  the  danger. 

3.  Determination  of  the  geographic  or  political  subdivisions 
exposed. 
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4.  Formulation  of  the  warning. 

5.  Determine  the  routing  of  the  message. 

6.  Prepare  the  message  into  paper  tape  form. 

7.  Transmission  of  the  warning  on  the  circuits. 

8.  Relay  of  warning  at  adjacent  stations  to  other  circuits 
when  watches  or  warnings  cross  state  boundaries. 

With  AFOS,  steps  5,  6,  7,  and  8  are  completely  automated. 

At  the  conclusion  of  the  formulation  (step  4)  a  single  key  entry 
enters  the  message  to  the  local  NWWS  circuit  and  then  computers 
at  adjacent  WSFO's  automatically  route  the  warning  onto  other 
NWWS  circuits.  The  time  required  to  carry  out  steps  3  and  4  is 
decreased  substantially  by  the  system's  message  composition 
capabilities,  including  preformat  of  warnings,  and  computer 
programs  to  assist  in  the  determination  of  the  specific  affected 
counties.  Steps  1  and  2  are  enhanced  by  the  automatic  display  of 
existing  weather  conditions  and  guidance  products  that  will  save 
valuable  time  for  the  forecaster  in  monitoring  and  watching 
rapidly  changing  mesoscale  weather  situations  and  in  analyzing 
the  potential  need  for  weather  warnings  or  special  forecasts. 

This  forecaster  assistance  plus  automatic  display  and  iden¬ 
tification  of  affected  areas  saves  additional  time  in  getting  the 
forecast  and  warning  to  the  public.  Depending  on  the  speed  with 
which  the  weather  situation  is  developing  and  its  extent  and 
complexity,  the  automation  of  these  functions  by  AFOS  can  result 
in  additional  warning  time  of  from  5  to  30  minutes.  These  min¬ 
utes  can  be  translated  into  additional  time  for  the  public  to  pro¬ 
tect  itself  and  its  property  from  potential  danger. 

Productivity 

AFOS's  contribution  to  increased  productivity  is  primarily 
in  taking  over  the  routine  mechanics  of  forecasting  duty  which 
were  discussed  in  more  detail  earlier  in  the  chapter  (map 
plotting,  data  posting,  message  composition,  etc.)  thus  freeing 
forecasters  to  devote  more  time  to  more  professional  tasks.  The 
value  of  increased  capacity  for  production  can  take  different 
forms,  e.g.,  analyzing  the  weather  in  finer  detail,  updating 
forecasts  more  frequently,  responding  to  the  public  for 
information,  or  in  meeting  requirements  for  improved  weather 
services.  It  is  difficult  to  express  this  value  in  quantitative 
terms.  However,  the  1976  PDP  chose  to  use  the  amount  of  time 
saved  as  a  measure  of  the  value.  Based  on  our  experience  to 
date,  we  are  confident  that  AFOS  will  provide  at  least  1  hour 
saving  per  forecaster  day  at  our  WSFO's  and  large  WSO's  and  about 
one  half  hour  per  Weather  Service  Specialist  day  at  the  WSO's. 

With  approximately  1,000  forecasters  and  1,500  service 
specialists,  the  value  of  staff  time  saved  by  AFOS  will  be  over 
$8M  per  year.  When  familiarity  with  AFOS  capabilities  becomes 
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fully  ingrained  in  the  work  patterns  of  the  staff  and  the 
teletypewriter  have  been  removed,  there  will  be  additional 
savings  in  staff  time. 

The  program  development  plan  projected  a  net  savings  of 
177  positions.  In  addition,  119  positions  would  be  reprogrammed 
to  support  AFOS.  Presently  118  positions  are  being  used  to  sup¬ 
port  the  AFOS  program,  and  have  been  allocated  to  maintenance, 
network  monitoring,  control,  and  management  functions. 

The  source  of  the  projected  positions,  including  the  positions 
saved  and  those  needed  for  AFOS  support,  is  the  automation  of 
NWWS  —  a  total  of  296  positions.  In  1976,  the  AFOS  PDP  iden¬ 
tified  218  NWWS  positions  in  service,  plus  78  more  required  for 
the  planned  expansion  of  the  NWWS  systems  to  the  remaining 
13  states. 

Given  AFOS's  present  level  of  implementation,  it  has  been 
possible  to  reduce  the  NWWS  operations  staff  to  118,  who  are 
being  cross-utilized  to  support  other  programs  as  well.  Once 
AFOS  is  fully  implemented  nationally,  the  NWWS  function  will  be 
eliminated  entirely.  The  expansion  of  NWWS  is  proceeding  as 
planned  without  additional  positions.  Thus,  a  total  of 
100  positions  have  already  been  saved  and  applied  to  AFOS,  and 
the  remaining  position  savings  will  be  achieved  as  planned. 

The  additional  freed  positions  have  already  been  anticipated 
in  implementing  the  NOAA  Weather  Radio  program  without  new 
positions  and  in  allocating  personnel  reductions  the  past  few 
years.  This  has  created  an  understaf f ing  situation  in  the  NWS 
that  urgently  needs  correction,  and  AFOS  provides  the  only  know 
solution. 

The  communications  savings  are  realized  through  the  con¬ 
centration  of  more  information  onto  fewer  circuits.  Due  to  the 
anticipated  delay  in  AFOS  implementation,  together  with  require¬ 
ments  for  higher  quality  facsimile,  we  have  discontinued  two  cir¬ 
cuits  ( FOFAX  and  NAMFAX)  in  favor  of  a  higher  performance  DIFAX 
system.  In  1984,  the  savings  will  be  realized  from  the  removal 
of  the  facsimile  and  teletypewriter  terminal  equipment.  We 
expect  our  original  estimates  ($2.6M)  will  hold  up,  although 
delayed  somewhat  in  time. 

Thus,  while  the  value  or  cost  avoidance  picture  may  not 
match  precisely  what  was  anticipated  in  the  PDP  5  years  ago,  they 
are  quite  similar.  In  fact,  even  at  this  preoperat ional  stage, 
we  are  already  realizing  significant  value  from  the  AFOS  system. 

FUTURE  SYSTEM 


With  regard  to  the  development  of  a  follow-on  system  to 
AFOS,  the  PDP  prepared  in  1976  recognized  that  the  AFOS  system 
was  the  beginning  of  an  era  in  which  data  processing  and  display 
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technology  would  be  used  to  enhance  weather  services  by  assisting 
the  forecaster  at  the  field  office  level  of  the  Weather  Service. 
Also,  future  needs  and  opportunities  for  improving  weather  serv¬ 
ices  would  arise  requiring  modifications  and  expansion  of  the 
original  system.  The  GAO  has  not  separated  original  requirements 
from  future  requirements  and  opportunities.  For  example,  while 
the  present  AFOS  system  handles  processed  radar  and  satellite 
information,  it  is  not  capable  of  interactively  integrating 
satellite  and  radar  data  nor  was  it  intended  to  do  so.  Research 
in  this  regard  is  now  being  pursued  in  NOAA  under  the  Prototype 
Regional  Observing  and  Forecasting  Service  program  approved  in 
1980  by  the  Congress.  The  earliest  time  an  advanced  capability 
of  this  type  could  be  introduced  in  the  field  would  be  about 
1990.  To  try  it  sooner  would  involve  high  risk  and  crash  effort. 
Since  the  design  of  such  a  system  is  at  a  very  early  stage,  no 
useful  estimates  of  its  cost  can  be  made.  The  value  attributed 
to  the  Director  of  the  NWS,  in  the  report,  was  not  made  in  any 
numerical  sense.  He  stated  that  the  cost  would  be  at  least  as 
great  as  that  of  the  present  system. 

Potential  Cost  of  AFOS  1982-1989 

The  GAO  comparison  of  operating  the  Weather  Service  with  and 
without  AFOS  throughout  the  1980's  is  misleading  and  erroneous. 

In  the  GAO  analysis,  it  was  assumed  that  AFOS  will  have  zero 
value  to  the  Weather  Service.  The  above  discussion  has  shown 
that  there  is  substantial  value  to  be  derived  from  AFOS  in  the 
preparation  of  warnings  and  forecasts  and  in  their  dissemination. 
In  addition  to  the  GAO  analysis,  the  costs  of  operating  AFOS  are 
overstated  by  applying  an  overhead  scheme  which  is  different 
than  the  one  used  in  NOAA.  GAO  also  assumes  that  the  FAA  cir¬ 
cuits  will  be  available  through  1989;  when  actually,  FAA  plans  to 
remove  the  circuits  with  their  modernization  program. 

CONCLUSIONS 


NOAA  does  not  agree  with  the  conclusion  that  AFOS  is  5  years 
behind  schedule  and  $22M  over  budget.  As  pointed  out  in  the 
above  discussion  of  the  AFOS  schedule  and  costs,  the  AFOS  program 
is  2  years  behind  the  schedule  set  forth  in  the  1976  PDP  with  the 
operational  date  being  end  of  FY  1982  when  AFOS  is  the  primary 
mode  of  operation  at  all  equipped  stations  and  not  the  latter 
date  of  September  1984  when  the  standby  teletypewriter  and  fac¬ 
simile  communications  are  removed. 

In  arriving  at  the  $22M  overbudget  conclusion,  GAO  misin¬ 
terpreted  by  $9. 4M  the  initial  new  budget  authority  and  the  total 
congressional  authorization  for  AFOS  through  FY  1980.  Also,  GAO 
was  inconsistant  in  not  applying  overhead  costs  to  both  plan  and 
expenditures.  When  properly  applied,  NWS  estimates  the  amount 
over  budget  to  be  of  the  order  of  $2M. 
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We  do  not  agree  with  the  estimated  savings  which  GAO  ascribes 
to  maintaining  the  present  system  through  1989  versus  using  AFOS 
operationally  while  developing  a  new  system  for  implementation  at 
that  time.  GAO  overstates  the  continuing  cost  of  AFOS  by 
including  overhead,  erroneously  assumes  FAA  circuits  will 
continue,  and  fails  to  attribute  value  to  the  interim  operation 
of  AFOS. 

With  reference  to  accounting  for  AFOS  costs,  the  NWS  has 
followed  an  approved  organizational  accounting  system.  Some  of 
the  staff  effort  should  have  been  attributed  as  a  direct  cost  to 
AFOS.  This  will  be  corrected. 

NOAA  does  not  agree  that  it  has  understated  the  future  costs 
of  AFOS.  We  contend  that  it  is  not  appropriate  to  include 
overhead  costs.  The  AFOS  program  has  little  impact  on  the  basis 
for  allocating  such  distributable  costs. 

RECOMMENDATION 


The  action  recommended  is  not  considered  necessary.  The  NWS 
will,  as  it  has  in  past,  comply  with  Department  of  Commerce's 
accounting  system  and,  therefore,  intends  to  apply  A-109 
procedures. 
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CHAPTER  3 

MANAGEMENT  OF  AFOS 


Scope  of  AFOS  Program  Management 

The  introduction  of  AFOS  into  the  National  Weatner  Service 
substantially  changes  tne  way  thousands  of  personnel  do  their 
work.  The  system  is  pervasive  and  directly  affects  almost 
evary  organizational  unit  across  the  country.  The  system  is 
very  different  from  the  conventional  centralized  ADP  facility 
staifed  and  operated  by  ADP  professionals.  It  is  a 
decentralized,  distributed  processing  system  in  whicn  the  local 
processors  serve  as  an  iriLimate,  interactive  extension  of  the 
field  forecasters  at  each  field  office.  At  the  same  time,  the 
processors  are  integral  elements  of  a  unified  tele¬ 
communications  and  information  storage  system  that  spans  the 
entire  country.  These  latter  functions  require  strong, 
centralized  operational  control. 


For  these  reasons,  the  management  arrangements  for  AFOS 
nave  to  consider  much  more  than  the  development  and  testing  of 
the  nardware,  software  and  support  elements  for  tne  distributed 
processing  system,  even  though  that  is  an  extremely  large  ana 
complex  task  for  which  few  if  any  precedents  exist.  The 
management  arrangements  also  have  to  motivate  and  facilitate 
tue  transition  to  a  totally  different  mode  of  operation  for  the 
majority  of  the  NWS  workforce,  and  also  alter  many  significant 
relationships  with  other  agencies  and  users.  All  this  has  to 
oe  accomplished  witnout  disruption  to  existing  services  and  tne 
activities  of  external  users,  concurrently  with  the  expansion 
and  improvement  of  service  functions,  and  witnout  any  increase 
in  NWS  staffing  levels.  Furtner,  it  had  to  be  accomplished 
within  an  organization  having  more  than  a  century-long  tra¬ 
dition  in  the  previous  mode  of  operation,  with  very  limited 
prior  exposure  to  computers  and  automation  technology  in  tne 
field  force.  Finally,  as  tne  program  progressed  througn 
various  development  stages,  problems  were  encountered  tnat 
required  management  adjustments. 


The  institution  and  management  arrangements  must  be 
responsive  to  the  above  situation.  These  arrangements  have 
evolved  during  the  course  of  the  program  in  response  to  the 
changing  emphasis  needed  at  different  stages,  actual  experience 
witn  the  arrangements  chosen,  and  recommendations  made  by 
consultants  retained  by  NWS  to  assist  in  areas  where  special 
expertise  was  available.  The  management  schemes  adopted  by  NWS 
were  innovative  and  deliberately  departed  in  some  ways  from 
classical,  text-book  project  structures  because  the  needs  of 
the  program  extended  far  oeyond  the  classical  problems  those 
structures  were  devised  to  address.  NWS  management  officials 
feel  that  the  general  thrust  of  GAO's  criticism  of  NWS 
management  reflects  a  distorted  emphasis  on  classical 
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"engineering"  system  development  as  practiced  in  development 
organizations  and  a  rather  shallow  appreciation  of  the 
real-world  problems  of  development  and  change  within  a  service 
organization  conducting  day-to-day,  time-critical  operations. 
The  priorities  necessarily  must  be  different. 

Nevertheless,  NOAA  recognizes  the  validity  of  most  of  the 
general  management  principles  cited  in  the  draft  report,  and  a 
conscientious  effort  has  been  and  is  being  made  to  apply  them 
effectively  in  the  management  of  AFOS.  In  fact,  in  the  initial 
phases  of  the  AFOS  program  the  entire  development  -  system 
analysis,  hardware,  software,  and  system  experimentation  and 
evaluation  was  organized  under  one  single  project  manager 
reporting  to  the  Office  of  the  Director  of  NWS.  The  project 
manager  was  an  experienced  systems  engineer.  It  was  during 
t»iis  period  that  many  of  the  decisions,  repeatedly  critized  by 
GAO,  on  software,  hardware  and  telecommunications  were  maae. 

In  other  words  purity  of  organization  does  not,  as  GAO  implies, 
necessarily  lead  to  success. 

In  hindsignt,  we  recognize  that  some  decisions,  actions 
and  inactions  resulted  in  problems  that  were  not  forseen  at  the 
time  and  might  have  been  avoided  had  another  patn  been 
selected.  In  general,  NOAA  does  not  agree  with  GAO  that  a 
large  fraction  of  the  problems  encountered  in  AFOS  were  caused 
mainly  by  the  lack  of  coherent  management,  but  rather  by  the 
sweeping  scope  and  inherent  complexity  of  the  program.  While  a 
different  management  approach  may  have  reduced  or  avoided  some 
proDlems,  it  would  have  made  other  kinds  of  problems  more 
li kely . 

Current  AFOS  Management  Structure 

NOAA  believes  that  the  NWS  has  learned  much  from  the 
problems  encountered  (and  the  successes  achieved).  A  much  more 
mature  program  structure  is  now  in  place,  and  the  technical 
knowledge  needed  to  operate  and  sustain  AFOS  has  been  broadly 
established  throughout  the  organization.  The  existence  of  past 
problems,  whetner  attributable  to  management  deficiencies  or 
not,  is  not  a  rational  basis  for  not  using  a  system  after  the 
problems  are  resolved.  *’ 

The  current  AFOS  management  structure  explicitly  addresses 
the  current  stage  of  transition  of  the  initial  system  from 
development  and  testing  into  operational  use.  There  is  a 
single  AFOS  Program  Manager,  an  SES  member  wno  reports  directly 
to  the  Director  of  the  NWS,  and  who  has  been  delegated  broad 
responsibility  and  authority  for  the  program.  In  apparent 
contradict ' on  to  the  draft  GAO  report,  all  staff  responsible 
for  development  of  AFOS  report  to  him  (many  through 
intermediate  supervisors) .  While  this  contradiction  may  result 
partially  from  adjustments  made  during  the  course  of  GAO's 
review,  it  nay  stem  partly  from  confusion  over  the  role  of 
field  personnel.  As  described  in  Chapter  2,  AFOS  includes  a 


APPENDIX  II 


APPENDIX  II 


powerful  and  versatile  capability  to  support  local  applications 
concurrently  with  the  telecommunications  and  other  functions. 
The  system  capability  for  local  applications  has  been  developed 
and  managed  centrally.  However,  many  of  the  specific  computer 
programs  (instructions)  to  carry  out  specific  applications  have 
been  developed  by  field  personnel  with  the  full  blessing  and 
authorization  of  central  management.  Most  of  these  appli¬ 
cations  o re  tailored  to  the  needs  of  local  service  offices,  and 
local  managers  have  been  delegated  responsibility  and  authority 
for  assigning  priorities  for  use  of  the  resources  allocated  to 
local  processing  within  the  overall  AFOS  system.  Central 
management  provides  and  enforces  written  guidelines  and 
standards  for  local  applications  programs,  and  maintains  a 
clear ingnouse  function  for  documentation  and  national  exchange 
of  applications  programs. 

The  AFOS  Program  Manager  has  the  authority  to  ma  ke 
decisions  and  establish  policies  which  bind  all  organizational 
elements  oi  NWS,  including  the  field  elements.  Obviously,  this 
must  be  done  judiciously  and  with  broad  participation  by  man¬ 
agers  of  tne  affected  elements  to  insure  that  continuing 
service  operations  are  not  disrupted.  It  is  true  that  the 
Directors  of  the  NWS  field  regions  don't  report  to  the  AFOS 
Program  Manager  and  are  not  formally  rated  by  him,  nor  do  the 
Director  of  the  National  Meteorological  Center  or  the  Directors 
of  the  other  three  National  Headquarters  Offices.  Such  an 
arrangeme.it  would  be  tantamount  to  redefining  the  Director  of 
NWS.  Nevertheless,  for  AFOS  program  matters  the  Director 
relies  on  the  AFOS  Manager  and  has  directed  the  other  senior 
managers  to  follow  his  leadership  unless  an  irreconcilable 
conflict  occurs  that  must  be  resolved  by  the  Director.  No 
problems  of  substance  have  occurred  with  this  arrangement  since 
it  was  instituted  in  March  of  1980. 

The  implementation  of  the  initial  AFOS  System  nas  been 
assigned  in  a  project  sense  to  a  single  project  manager  under 
the  AFOS  Program  Manager.  All  personnel  involved  in  final 
development  and  testing,  network  monitoring  and  control, 
training,  procedural  development  and  documentation  report  to 
nim  as  part  of  a  Transition  Task  Team.  The  Team  compromises  a 
Development  Task  Group  and  an  Operations  Task  Group,  each 
headed  by  an  StS  member  at  the  Laboratory  and  Division  Director 
level.  The  Operations  Task  Group  includes  the  entire,  newly 
created  AFOS  Operations  Division  that  will  be  responsible  for 
continuing  management  of  all  aspects  of  AFOS'  operations 
following  the  formal  commissioning  of  AFOS.  All  managers  in 
NWS  field  organization  who  are  responsible  for  AFOS,  down  to 
the  field  station  level,  take  direction  on  AFOS  matters  from 
the  AFOS  Operations  Manager  wno  heads  the  division. 

The  maintenance,  logistics  and  facilities  support  for  AFOS 
is  integrated  with  all  similar  functions  in  NWS,  both  at  the 
headquarters  level  in  the  Office  of  Technical  Services  where 
the  overall  programs  are  managed  and  in  the  field  organization 
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where  technicians  for  local  sites  and  service  areas  are 
located.  As  AFOS  becomes  operational,  a  growing  fraction  of 
the  effort  in  tnese  programs  shifts  to  support  of  AFOS.  The 
AFOS  Operations  Manager  also  is  formally  designated  Deputy 
Director  of  the  Office  of  Technical  Services,  and  therefore  nas 
actual  line  autnority  over  the  key  operational  elements  sup¬ 
porting  AFOS  as  part  of  nationally  integrated  support  programs. 

The  AFOS  Program  Manager  also  has  full  responsibility  and 
authority  for  future  system  development,  and  all  staff  involved 
in  this  effort  report  to  him.  He  also  serves  as  chairman  of 
the  formal  AFOS  Cnange  Management  Board  at  the  Office  Director 
level  which  is  chartered  by  the  Director,  with  broad  delegation 
of  authority,  to  establish  all  policies  and  mechanisms  nec¬ 
essary  to  control  and  promote  orderly  improvements  in  the 
system  and  its  operations.  To  date,  mechanisms  have  been 
chartered  and  estaoxished  by  the  Change  Management  Board  for 
configuration  control;  addition,  deletion  and  change  in 
priority  of  AFOS  products;  and  local  applications  development 
and  exchange. 

A  plan  has  been  developed  under  the  leadership  of  the  AFOS 
Program  Manager  for  completing  development  and  implementation 
of  the  initial  AFOS  system;  introducing  enhancements  to  the 
system  (e.g.,  to  equip  with  remote  terminals  the  smaller  NWS 
stations  not  planned  to  have  stand-alone  AFOS  computers,  and 
withdrawing  temporary,  backup  teletypewriter  and  facsimile 
systems) ;  and  developing  major  system  improvements  that 
incorporate  data  types  and  functional  capabilities  never 
planned  for  the  initial  AFOS  system.  This  plan  covers  the  time 
period  into  the  1990's,  with  proportionally  less  specific 
technical  and  operational  detail  in  distant  years,  but 
sufficient  to  define  the  overall  strategy,  guide  budget  and 
program  development  activities,  and  establish  priorities. 

NOAA  believes  that  the  existing  AFOS  management  arrange¬ 
ments  are  well-conceived  and  they  consider  not  only  the  points 
raised  by  GAO,  but  many  other  factors  not  even  recognized  by 
GAO,  some  of  them  much  more  important  to  the  success  of  the 
AFOS  Program.  The  NWS  management  personnel  at  all  levels  in 
the  organization  are  no  longer  inexperienced  in  ADP  system 
development  and  management,  even  if  that  were  partially  true  at 
an  earlier  time.  NOAA  noces  the  success  NWS  has  achieved  in 
the  difficult  task  of  developing  and  introducing 
interactive ,distr ibuted  processing  technology  into  a  service 
operations  environment. 

Comments  on  GAO  Views 


The  specific  comments  on  AFOS  management  by  the  GAO, 
unfortunately,  contain  misconceptions,  over-simplication  of 
cause  and  effect.  We  hope  that  the  foregoing  summary  of  the 
current  AFOS  management  arrangements  dispels  most  of  the 
confusion  and  clearly  distinguishes  facts. 
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In  particular,  the  current  arrangement  includes  the  Key 
management  principles  tnat  GAO  claims  we  have  not  followed, 
e.g.  management  focus,  clear  lines  of  authority,  sufficient 
authority,  balance  between  centralized  and  decentralized 
authority,  use  of  project  managers  and  project  management 
disciplines,  and  use  of  comprehensive  plans. 

It  is  difficult  to  answer  the  historical  charges  one  by 
one  without  allocating  many  paragraphs.  Suffice  it  to  say  tnat 
we  consider  it  unlikely  that  the  existing  system  could  have 
been  brought  to  the  successful  stage  of  development  now 
achieved,  if  the  program  management  were  as  lacking  as  claimed. 
NOAA  admits  that  unforeseen  problems  did  occur  in  the  program, 
but  they  have  been  identined  and  corrected,  and  this  success 
is  a  better  measure  of  future  expectations. 

Contrary  to  the  broau  generalizations  by  GAO,  NOAA  applied 
appropriate  approacnes  to  managing  large  projects.  Certainly, 
not  all  approaches  were  applied  because  some  are  alternatives 
to  others.  Similarly,  concerning  our  use  of  contractors  and 
consultants,  not  all  recommendations  were  automatically 
accepted  and  implemented.  NWS  procured  the  entire  hardware 
system  through  contractors,  obtained  major  pieces  of  software 
througn  contractors,  and  ootained  and  applied  a  great  deal  of 
useful  advice  from  contractors.  The  performance  of  the  system 
validates  these  uses  of  contractors.  It  is  illogical  to  argue 
that  each  and  every  recommendation  of  consultants  should  be 
followed  in  detail.  In  fact,  it  would  be  impossible  because 
consultants  don't  always  agree.  No  contractor  has  the  breadth 
and  completeness  of  knowleuge  about  program  issues  that  the 
agency  possesses,  and  all  recommendations  must  be  evaluated  by 
the  agency  in  the  light  of  all  considerations.  (For  example, 
an  unquoted  corollary  recommendation  to  one  of  those  quoted  in 
the  GAO  report,  by  the  same  consultant,  suggested  that  we  add 
immediately  a  dozen  software  experts  in  real-time  systems 
analysis  in  order  to  complete  a  critical  job  in  the  succeeding 
few  months,  a  desirable  but  totally  impractical  action.) 

The  GAO  report  repeatedly  stresses  the  lack  of  experience 
of  NWS  with  ADP  system  development,  criticizes  on  page  23  the 
expertise  of  NWS  managers,  urges  contracting  out  major  ADP 
system  development,  and  then  finishes  by  asserting  the 
concurrence  of  the  Director  of  NWS  with  (an  unspecified  portion 
of)  the  GAO  position.  In  truth,  the  Director  of  NWS  does 
concur  in  the  appropriate  use  of  contractors.  However,  he 
reports  having  encountered  far  more  problems  resulting  from  ADP 
specialists  who  do  not  understand  weather  services  and 
government  procedures.  The  management  staff  must  appreciate 
both  weather  services  and  systems  development,  and  persons  who 
are  recognized  among  the  top  national  experts  in  both 
disciplines  are  rare  or  non-existent. 
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CONCLUSIONS 

Referring  to  the  summary  conclusions,  we  believe  that  GAO 
has  exaggerated  the  influence  of  management  deficiencies  on 
AFOS  development  problems,  and  has  failed  to  understand  fully 
the  reasons  for  the  management  approaches  used.  That  not  all 
problems  were  foreseen  cannot  be  attributed  primarily  to 
management,  but  results  largely  from  the  complexity  of  the 
program  and  the  difficulty  everyone  has  in  predicting  the 
future.  The  only  real  cricerion  for  evaluating  management  is 
the  degree  of  success  of  tne  program,  and  the  AFOS  System 
successfully  performs  all  essential  functions  and  is  used  to 
support  routine  service  operations  even  today.  NWS  now  has  a 
trained  and  experienced  development  team,  operating  force  and 
management  group.  These  accomplishments  should  be  the  basis 
for  judging  the  accuracy  ana  relevance  of  the  GAO  assertions. 

RECOMMENDATIONS 

The  NWS  already  has  implemented  the  specific  GAO 
recommendations  that  are  applicable  to  the  existing  system,  and 
furture  plans  are  in  concurrence  with  those  referring  to  the 
new  system.  In  particular,  the  following  actions  have  been 
ta  Ken: 


1.  An  AFOS  project  management  office  has  been 
established  and  all  development  personnel 

have  been  assigned  to  that  office  on  a  full-time 
basis. 

2.  A  project  manager  has  been  appointed  with  clear 
authority  for  the  project. 

For  development  of  the  new  system,  NWS  plans  include: 

3.  Selection  and  enforcement  of  standard  soft¬ 
ware  development  procedures,  including  documen¬ 
tation  and  testing  for  the  new  system,  and 

4.  Contracting  system  development  activities. 
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CHAPTER  4 
AFOS  SOFTWARE 


Software  Capability 

Advances  in  the  science  or  meteorology  over  the  years  have 
been  substantial.  Forecasters  today  have  enhanced  analytical 
techniques,  computer  models,  and  computer-generated  guidance  to 
aid  them  in  the  formulation  of  their  products  and  forecasts. 
But,  forecasts  are  still  in  part  the  result  of  the 
individualized  application  of  experience,  the  complex 
assimilation  of  current  and  historical  weather  information,  and 
in  some  instances  the  "gut  feeling"  of  the  forecaster.  This 
variability  in  procedure  coupled  with  a  workforce  generally 
unfamiliar  with  automation  technologies  impeded  the  development 
of,  and  agreement  on,  detailed  requirements  in  areas  where 
personnel  interact  with  the  system.  On  the  other  hand,  many 
system  software  requirements  were  originally  specified  in  the 
design  stages  of  the  AFOS  program  (mid-1970's)  and  have 
remained  throughout. 

Originally  the  AFOS  software  development  involved  designers 
and  users  in  an  iterative  design,  test,  evaluate,  and  modify 
process  that  attempted  to  capture  and  define  the  more  subtle 
software  requirements.  AFOS  software  development  began  in  the 
mid-1970's  and  was  scheduled  to  take  place  over  the  time  frame 
of  several  years  and  not  the  »0  days  the  GAO  contends.  This 
form  of  development,  if  properly  managed  and  controlled,  often 
yields  the  most  satisfactory  systems  performance.  As  the  GAO 
points  out,  the  NWS  did  encounter  difficulties  managing  this 
phase  of  the  software  development  which  affected  the  overall 
software  development  timetauie.  But  this  process  yielded  a 
system  whose  functional  performance  and  capabilities  could  be 
effectively  used  by  field  personnel  and  one  which  has  been 
widely  accepted  throughout  tne  NWS. 

AFOS  software  meets  the  original  objectives  as  listed  in 
the  AFOS  Program  Development  Plan  and  subsequently  determined 
by  interaction  with  operational  personnel.  As  described  in 
detail  in  Chapter  2,  AFOS  receives  information  from  the 
telecommunications  subsystem  and  stores  it  for  subsequent  use 
by  the  forecasters;  the  system  retrieves  stored  information  for 
display  in  various  useful  ways;  and  it  facilitates  the 
composition  and  automatic  transmission  of  messages  and  carries 
out  many  meteorological  and  hydrological  applications  and 
services.  The  AFOS  software  now  in  actual  field  use 
accomplishes  these  essential  tasks  reliably,  rapidly,  and  makes 
effective  use  of  the  computer,  telecommunications  resources, 
and  other  system  features. 
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Software  Development 


A  point  that  cannot  be  overstated  is  that  AFOS  is  a  complex 
system.  AFOS  software  is  also  complex  and,  as  the  GAO  states, 
is  tightly  integrated.  While  tne  tightly  coupled  or  integrated 
software  is  a  problem  to  those  who  must  design  subsequent 
changes  in  AFOS,  it  does  not  affect  the  operational  user.  Even 
though  the  software  performance  is  entirely  satisfactory,  NWS 
intends  to  make  a  few  structural  changes  to  the  software  to 
facilitate  improvements  as  operational  experience  is  gained. 
These  changes  would  also  incorporate  additional  computer  memory. 

The  NWS  has  demonstrated  effective  software  development 
management  practices.  For  example,  in  April  1980  the  entire 
asychronous  communication  subsystem,  representing  an  additional 
one-third  of  the  real-time  computer  code,  was  successfully 
integrated  into  the  existing  software,  debugged,  and  tested. 

This  fact  directly  contradicts  the  GAO  contention  that  "from 
1978  to  the  present  the  developers  have  been  reducing  system 
capability  in  an  effort  to  acnieve  a  stable  system"--an 
unsupported  and  false  allegation.  The  GAO  report  lists  other 
software  problems  such  as:  the  communications  subsystem  does 
not  perform  acceptably,  more  tnan  one  forecaster  cannot  use  the 
system  at  a  time,  only  one  software  back-up  routine  works 
reliably  and  message  composition  failures  cause  the  loss  of 
prepared  messages.  Currently  the  AFOS  software  permits  the 
satisfactory  use  of  all  consoles  at  each  field  site  (the 
largest  station  has  8  consoles) ,  and  message  composition 
recovery  is  fully  operable  and  prevents  the  loss  of  text.  The 
system  back-up  capabilities  as  oescribed  in  Chapter  2  are 
tested  and  in  use  today,  and  the  AFOS  telecommunication  system 
functions  satisfactorily  as  described  in  Chapter  5. 

In  mid-1979  NWS  managemenc  determined  that  the  developers 
were  having  technical  difficulty  in  completing  development  and 
tests  of  the  system  and  there  was  a  need  to  strengthen  overall 
program  management.  The  new  program  management  immediately 
instituted  classically  accepted  development  and  testing 
disciplines.  An  interim  software  change  management  process  was 
invoked  and  utilized  until  tne  spring  of  1981  when  the  NWS 
formally  instituted  an  AFOS  Change  Management  Board  with  a 
specific  charter  and  delegations  of  responsibility  and 
authority  from  the  Director,  NWS. 

Prior  to  mid-1979  software  was  insufficiently  documented, 
but  GAO's  strong  criticism  of  documentation  is  not  valid  now. 
Hardware  documentation  was  fully  established  by  the  AFOS 
hardware  contractor.  An  initial  level  of  software 
documentation  was  establisheu  by  an  intense  5-month  effort  of 
the  entire  software  development  staff.  In  1980,  the  NWS 
contracted  with  a  private  contractor  to  develop  a  formal  set  of 
software  documentation.  This  documentation,  containing  more 
than  8,000  pages  in  14  separate  bound  volumes,  will  start  being 
delivered  to  NWS  in  September  1981. 
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Software  Test  Progr*™ 

The  AFOS  test  program  was  re-established  on  a  more  firm 
technical  base  and  systematically  followed.  Exactly  how  to 
accomplish  this  resulted  from  extensive  deliberations  internally 
and  with  private  consultants.  me  general  consensus  was  that 
the  capabilities  and  functional  performance  of  the  system  had 
evolved  to  a  satisfactory  level,  but  system  stability  was 
inadequate.  A  path  was  chosen  that  made  maximum  use  of  the 
iterative  development  process  described  earlier.  To  the 
developers  this  was  translated  to  the  following  conclusions: 
tne  fundamental  algorithms  and  processes  being  performed  by  the 
software  modules  were  satisfactory;  the  interactions  of  the 
software  modules  alone  and  in  aggregate  numbers  were  most 
probably  the  cause  of  system  instability;  lesser  softw're 
defects  or  "bugs"  would  be  uncovered  and  corrected. 

A  highly  specialized  test  methodology  was  then  designed  and 
implemented.  A  test  structure  was  laid  over  the  existing  system 
which  provided  accountability  and  intermediate  goals.  A  test 
management  structure  was  implemented  at  that  time  using 
individuals  who  had  no  previous  investment  in  the  development 
process.  This  plan  of  attacri  was  reviewed  and  endorsed  by  a 
private  contractor. 

The  test  program  initiated  in  early  1980,  while  not  fully 
ortnodox  in  its  design,  was  well  considered  and  has  been 
successful.  At  that  time  system  stability  was  relatively  poor 
with  stations  experiencing  system  interruptions  almost  hourly 
which  required  6  to  10  minutes  to  correct.  The  stability 
currently  being  reported  by  the  field  stations  is  less  than  one 
int»  ruption  per  shift  and  requiring  only  2  to  5  minutes  to 
correct.  This  is  a  fully  acceptable  level  for  operational  use 
as  demonstrated  by  the  ability  of  two  of  the  four  contiguous 
regions  to  use  AFOS  as  the  primary  tool  for  the  delivery  of 
weatner  services. 

CONCLUSIONS 

NOAA  does  not  agree  wicn  the  GAO  conclusions  that  software 
developed  for  AFOS  has  serious  problems  which  can  only  be 
resolved  by  a  complete  redesign.  A  systematic  NWS  test  program 
has  validated  the  software  design.  The  current  actual  use  of 
the  AFOS  software  in  day-to-day  field  operations  demonstrates 
satisfactory  performance.  The  NWS  is  planning  to  make  some 
limited  improvements  to  tne  software  in  the  future.  We  intend 
to  make  appropriate  use  of  contractors  for  this  effort. 

RECOMMENDATION 


NOAA  does  not  agree  with  the  recommendation  to  redevelop 
AFOS  software  as  part  of  a  complete  redesign  and  redevelopment. 
The  existing  software  meets  all  essential  requirements  for  field 
use . 


108 


APPENDIX  II 


APPENDIX  II 


CHAPTER  5 

AFOS  TELECOMMUNICATIONS 
AFOS  Telecommunication  Design 


For  decades  the  NWS  nas  been  using  multiple  low  speed 
teletype  and  facsimile  circuits  to  deliver  all  of  the  alpha¬ 
numeric  and  graphic  information  the  forecaster  needs  to 
generate  forecast  products.  A  single  AFOS  telecommunication 
system  is  able  to  transmit  all  of  these  different  a lpha-numer ic 
and  graphic  products  reliaoiy  enough  for  forecasters  to  continue 
to  provide  warning  and  other  weather  services  to  the  public. 

In  addition,  AFOS  has  demonstrated  improvement  to  other  aspects 
of  NWS  operations.  AFOS  provides  warnings  and  other  service 
products  to  the  public  and  other  users  via  NWWS.  NOAA  does  not 
intend  to  provide  basic  meteorological  data  and  information  to 
private  users  and  other  meteorologists  on  AFOS,  but  will  provide 
separate  communication  facilities  for  this  purpose. 

AFOS  communication  speeas  are  an  order  of  magnitude  faster 
than  existing  teletype  speeas  thus  permitting  the  reliable 
delivery  of  weather  information  at  significantly  earlier  times. 
This  affords  forecasters  valuable  additional  time  to  analyze  and 
prepare  warning  messages.  Tne  AFOS  system  automatically 
transmits,  stores,  and  updates  a  site's  local  data  base  without 
human  intervention  thus  acnieving  even  greater  productivity  of 
NWS  personnel. 

Contrary  to  the  GAO  interpretation,  AFOS  communication 
circuits  are  aligned  with  the  decentralized  organization  of  the 
NWS.  Each  Regional  Distribution  Circuit  (RDC)  is  supplied  the 
same  meteorological  data  by  the  System  Monitoring  and 
Coordination  Center  (SMCC)  in  Suitland,  Maryland.  All  Weather 
Service  forecast  offices  (WSFO's)  within  a  Region,  are  connected 
on  their  own  RDC.  Each  forecast  office  has  associated  with  it  a 
number  of  Weather  Service  Offices  (WSO'S)  that  are 
interconnected  directly  to  WSFO's  by  State  Distribution  Circuits 
(SDC)  which  is  in  alignment  with  the  area  forecast  management 
structure  within  a  state. 

The  data  and  information  flowing  into  and  out  of  the  AFOS 
System  is  part  of  the  world  wide  interchange  of  weather 
information  that  taxes  place  at  the  National  Meteorological 
Center  (NMC)  in  Suitland,  Maryland.  The  NWS  has  operated  this 
communication  hub  reliably  for  decades. 

The  selection  of  the  AFOS  communications  loop  architecture 
was  the  result  of  extensive  analysis  by  the  NWS,  private 
companies,  and  the  National  Bureau  of  Standards  in  the 
mid-1970's.  A  communication  network  did  not  exist  that  suited 
AFOS'  purpose.  As  the  AFOS  communication  system  has  been 
implemented  and  tested,  numerous  improvements  have  evolved  that 
have  increased  overall  performance  and  reliability.  The  testing 
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has  been  performed  largely  by  the  NWS,  but  a  private  contractor 
was  assigned  to  review,  test,  and  evaluate  the  system  also. 

Reliabili ty 

All  AFOS  communication  circuits  are  installed  and 
operating.  Presently  the  Si4CC  is  in  its  final  configuration, 
having  recently  completed  a  series  of  engineering  and 
validation  tests.  Each  RDC  is  being  driven  from  its  assigned 
SMCC  computers.  SMCC  operations  are  meeting  service 
requirements  for  reliability  and  timeliness  of  data  flow 

As  the  evolution  of  the  telecommunication  system  reached 
the  levels  of  reliability  and  stability  the  NWS  needs,  the 
RDC's  were  implemented  starting  in  1980,  and  each  region  began 
the  process  of  transition  to  full  AFOS  operations.  Since  that 
time  the  stability  and  reliaoility  of  AFOS  telecommunication 
has  continued  to  improve  as  management  and  operational 
procedures  have  been  developed  and  refined. 

The  reliability  of  AFOS  communications  depends  on  several 
factors,  the  principal  ones  being  the  dedicated  telephone  links 
connecting  each  node  (WSFO) ,  secondly  the  sites'  computer 
controlled  and  managed  synchronous  communication  subsystem,  and 
thirdly  the  various  backup  modes  designed  into  the  system.  All 
these  components  have  been  tested  separately  and  as  an 
integrated  system.  The  performance  of  the  entire  system  in 
actual  operations  under  all  types  of  environmental  conditions 
is  satisfactory  for  NWS  operational  use. 

The  telecommunication  system  back-up  modes  incorporate  the 
following  automatic  and  operator  initiated  processes:  (1)  the 

AFOS  communication  computers  use  an  industry  accepted 
communication  protocol  that  incorporates  automatic  error 
detection  and  retransmission  features.  (2)  Data  is  routed 
bi-directionally  around  each  RDC  thus  ensuring  that  should  a 
particular  node  or  telephone  line  fail,  all  other  nodes  receive 
data.  (3)  In  the  event  that  tne  connecting  link  between 
adjacent  nodes  fails  the  communication  computers  automatically 
reconnect  with  each  other  using  standard  dial-up  circuits. 

(4)  Should  a  node  experience  a  catastrophic  failure,  the 
computers  at  adjacent  AFOS  noues  can  be  directed  by  station 
personnel  to  "dial  around"  the  failed  node.  (5)  As  described 
in  Chapter  2  either  of  the  two  computers  located  at  a  node  can 
perform  all  telecommunication  functions  thus  providing  an 
additional  level  of  back  up.  (6)  the  telecommunication 
subsystem  for  either  computer  has  back-up  equipment  also.  All 
back-up  modes  have  been  extensively  tested  and  perform 
satisfactorily.  In  contrast  to  this  level  of  sophistication, 
the  teletypewriter  circuits  used  today,  which  are  also  loop 
networks,  contain  none  of  these  back-up  features.  If  a 
teletypewriter  loop  is  broken,  communications  cease. 
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A  significant  disagreement  exists  between  the  NWS  and  GAO 
on  the  level  of  support  required  for  telecommunication.  Given 
the  overall  current  high  level  of  stability  of  the  AFOS 
hardware,  software,  and  communication  lines,  the  amount  of 
operator  intervention  or  attention  to  maintain  systems 
operation  is  currently  reporteo  at  less  than  5  minutes  per  8 
hour  shift.  This  is  significantly  less  than  the  time  required 
to  service  the  multiple  teletypewriter  and  facsimile  equipment 
at  each  site.  It  is  also  significantly  less  time  than  that 
whicn  the  GAO  observed  as  late  as  January  of  this  year. 

As  described  earlier,  the  bMCC  plays  a  vital  role  in  AFOS 
communication.  It  must  interface  with  NMC  and  simultaneously 
drive  the  four  RDC's.  In  addicion,  SMCC  monitors  the  status  of 
the  RDC’s,  notifies  station  personnel  of  malfunctions,  and 
advises  them  on  the  corrective  action  to  be  taken.  At  the 
point  in  time  when  the  GAO  inspected  SMCC  it  had  neither  been 
reconfigured  to  drive  four  RDC's  nor  had  it  undergone  the 
planned  engineering  and  validation  tests  scheduled  for  it. 

Hence  the  GAO's  pessimistic  view  on  the  adequacy  of  SMCC  is 
understandable.  Since  March  of  this  year  however,  scheduled 
tests  have  been  conducted  that  verified  the  performance  and 
reliability  of  the  SMCC  computers.  The  stability  of  the  SMCC 
systems  is  now  extremely  high.  Defects  uncovered  during 
testing  have  been  corrected  and  retested.  The  SMCC  is  driving 
the  four  RDC’s  with  live  meteorological  data  from  the  National 
Meteorological  Center. 

In  January  of  this  year,  tne  NWS  conducted  a  major 
engineering  test  of  the  AFOS  system  on  a  test  network  that 
interconnected  several  sites  in  each  region.  Station  personnel 
were  instructed  to  use  their  AFOS  systems  to  the  degree  their 
training  and  experience  permitted.  This  was  a  test  of  the  AFOS 
system  in  a  near-operational  setting  under  the  random  variables 
of  environment,  nationwide  communication,  and  operator 
interaction.  It  was  not  a  test  of  NWS  service  operation  using 
AFOS.  Perhaps  the  NWS  did  not  adequately  explain  the  purpose 
of  these  tests,  nonetheless  tne  GAO  has  incorrectly  drawn 
conclusions  from  NWS  test  briefings  j..d  reports  concerning  the 
suitability  of  the  AFOS  system  to  perform  satisfactorily  in 
operational  use.  There  were  only  two  significant  engineering 
defects  discovered  during  the  Systems  Operations  Test.  One  had 
to  do  with  delayed  transmissions  from  NMC  to  SMCC,  the  other, 
the  presence  of  unnecessary  traffic  on  the  test  loop.  The 
latter  problem  was  easily  corrected  and  retested.  The  other 
defect  has  been  corrected  as  part  of  the  NMC-SMCC  validation 
tests. 
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Telecommunication  Alternatives 


The  NWS  has  been  examining  the  current  status  of  the 
telecommunication  technologies.  Apparently  the  GAO  has 
interpreted  this  effort  as  a  signal  that  the  NWS  did  not 
believe  the  design  of  the  AFOS  telecommunication  system  to  be 
viable.  The  NWS  has  been  using  private  contractors  for  the 
past  year  to  assess  the  current  state  of  telecommunications 
technology  from  two  viewpoints:  first  to  identify  an 
alternative  system  that  could  be  implemented  should  testing 
reveal  AFOS  communication  to  be  unreliable  and,  second,  to 
identify  trends  in  the  telecommunication  technology  in 
anticipation  of  the  development  of  the  next  generation  system. 
The  former  was  a  programmatic  plan  to  ensure  continuous 
implementation  of  AFOS;  however,  this  study  reveals  there  is  no 
off-the-shelf  system  to  replace  the  AFOS  communication  systems 
economically.  The  latter  is  a  continuation  of  the  NWS  practice 
of  conducting  periodic  technology  assessments.  We  feel  these 
are  prudent  and  responsible  programmatic  and  agency  planning 
activ it ies . 

CONCLUSIONS: 

The  GAO  argues  that  AFOS  telecommunication  is  not 
appropriate  for  meeting  NWS  needs,  is  inflexible,  unreliable, 
and  conflicts  with  the  NWS  organizational  philosophy.  NOAA 
cannot  agree  with  these  conclusions  and  has  demonstrated,  by 
botn  engineering  tests  and  actual  operational  use,  satisfactory 
and  reliable  performance  of  tne  telecommunication  system  in 
both  its  normal  and  backup  nodes.  The  NWS  agrees  with  GAO  t h\« 
AFOS  communication  is  the  least  costly  of  other  possible 
designs.  The  NWS  concludes  tnat  the  low  cost  of  AFOS 
communication  coupled  with  the  proven  satisfactory  performance 
yields  a  truly  cost-effective  telecommunication  system. 

RECOMMENDATION 

NOAA  does  not  agree  with  tne  GAO  recommendation  that  the 
AFOS  telecommunciation  system  oe  replaced  with  an  as  yet 
undefined  and  unproven  alternative  system.  The  present  AFOS 
system  is  performing  satisfactorily  and  any  effort  to  replace 
it  now  would  unnecessarily  delay  implementation  of  AFOS,  add 
significant  unnecessary  cost  and  prevent  the  NWS  from  utilizing 
a  very  important  and  effective  tool  in  the  discharge  of  its 
public  service  responsibility. 
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CHAPTER  6 
AFOS  HARDWARE 


Current  Status 


The  AFOS  hardware  consists  of  minicomputers,  display 
consoles,  printers,  magnetic  discs,  and  telecommunication 
equipment.  This  equipment  was  competitively  procured  against 
a  technical  specification  prepared  by  the  NWS.  The  hardware 
specification  established  requirements  for  system  functional 
capabilities,  maintainability,  and  reliability.  This 
procurement  also  required  the  contractor  to  perform  analysis 
and  specify  logistic  levels  ana  geographical  distributions  that 
would  ensure  satisfactory  and  reliable  hardware  maintenance. 

In  addition,  the  contractor  was  to  provide  NWS  specified 
computerized  automatic  test  equipment  and  the  necessary 
computer  diagnostic  programs  to  be  used  in  the  AFOS  repair 
depot  in  Kansas  City,  MO. 

Acceptance  of  the  hardware  systems  was  based  on  a 
contractor  generated  and  government  approved  test  program  that 
addressed  systems  performance,  reliability,  and  maintainability. 
Performance  tests  examined  eacn  required  capability  of  the 
system  while  the  hardware  was  subjected  to  specified  extremes 
of  electrical  power  and  ambient  temperatures  and  humidity. 
Hardware  reliability  tests  for  AFOS  are  basically  equipment 
endurance  tests  conducted  over  a  statistically  significant 
period  of  time  based  on  Mil-Spec  reliability  test  procedures. 

The  equipment  maintainability  tests  were  established  by  the 
contractor  and  performed  by  NWS  electronic  technicians.  These 
tests  demonstrated  that  NWS  factory  trained  technicians  using 
contractor  provided  maintenance  manuals,  specified  test 
equipment,  and  diagnostic  aids  were  capable  of  restoring 
equipment  operations  within  specified  times. 

The  AFOS  equipment  passed  all  required  tests  for 
performance,  reliability,  and  maintainability.  All  other 
contracted  deliverables  were  similiarily  accepted  based  upon 
approved  test  or  inspection  plans. 

The  AFOS  hardware  installation  began  in  1978  and  was 
completed  in  1981.  As  each  system  was  installed  NWS  maintenance 
and  logistic  reporting  programs  were  initiated  to  monitor  and 
analyze  significant  aspects  o£  the  hardware's  reliability, 
maintenance  and  performance  tnroughout  its  lifetime.  The 
equipment  is  expected  to  meet  the  initial  contract  acceptance 
criteria  throughout  its  use,  and  system  maintenance  costs  to 
remain  within  planned  levels. 
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AFOS  Logistics  System 

The  AFOS  logistics  system  aas  effectively  supported  AFOS 
operations  for  some  time  now.  It  includes  approximately  10,000 
Lowest  Repairable  Units  (LRU's)  i.e.,  repairable  sub-assemblies 
and  parts  and  about  30,000  non-repa irable  parts  that  support 
approximately  45,000  LRU's  in  daily  use  at  all  AFOS  sites. 

(The  number  of  non-repairable  parts  in  daily  use  exceeds  one 
million.)  Contractor  guaranteed-mean-t ime  between  failures 
projected  approximately  7,000  LRU  failures  per  year.  Currently 
NWS  is  recording  actual  failure*  rate  of  about  3,000  per  year. 
The  problems  with  the  flow  and  availability  of  spare  parts  have 
been  resolved.  AFOS  maintenance  and  logistics  programs  have 
gone  through  exhaustive  and  repeated  evaluation  by  NWS  and  are 
more  than  adequate  to  support  AFOS  operations.  The  AFOS 
maintenance  and  logistic  management  and  support  activities  are 
in  the  forefront  of  technology  and  are  effective  in  the 
operational  support  of  the  AFOS  program. 

The  AFOS  equipment  since  installation  began,  continues  to 
meet  and  in  most  instances  exceed  specifications  for 
performance,  reliability,  and  maintainability.  The  original 
estimates  of  the  need  for  65  additional  maintenance  technicians 
made  in  the  mid-1970's  are  still  valid  today.  Of  course  the 
technician  salaries  have  increased  substantially.  The  original 
projections  on  the  number  of  spare  parts  are  still  valid,  but 
inflation  has  increased  the  cost  of  spare  parts. 

Contrary  to  the  position  or  the  GAO,  the  NWS  has  through 
experience  established  effective  policies  that  determine 
whether  or  not  to  maintain  its  varied  equipment  systems  at 
current  manufacturers'  equipment  revision  levels.  Many  of  the 
manufacturers'  revisions  are  intended  to  reduce  manufacturing 
costs  or  modify  certain  performance  characteristics  and  not 
necessarily  correct  specific  problems.  The  NWS  closely 
monitors  AFOS  vendor  revisions  and  has  selectively  incorporated 
revisions  to  correct  specific  problems.  In  most  instances 
these  problems  were  identified  during  procurement  acceptance 
tests  or  contract  warranty  periods.  The  costs  to  correct  them 
are  normally  borne  by  the  AFOS  contractor. 

A  point  of  confusion  in  the  GAO's  argument  is  that  hardware 
and  software  improvements  maae  by  the  minicomputer  manufacturer 
cannot  be  incorporated  into  che  system  without  considerable 
cost  and  effort.  It  has  been  the  practice  of  the  computer 
trade  in  general,  and  the  manufacturers  of  AFOS  equipment  in 
particular,  to  ensure  that  any  manufacturing  hardware  revisions 
are  fully  compatible  with  its  original  system  in  form,  fit,  and 
function.  This  means  that  one  need  not  incorporate  every 
previous  revision  in  order  to  take  advantage  of  a  later  one 
that  may  correct  a  specific  defect. 
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Modifications  have  been  installed  either  by  the  contractor 
to  correct  latent  defects  or  by  the  NWS  to  modify  performance 
characteristics.  Equipment  modification  is  a  routine  type  of 
activity  in  all  maintenance  programs.  NWS  has  considerable 
experience  in  the  area  of  equipment  maintenance  and  modification 
since  it  maintains  more  than  7000  pieces  of  equipment  that 
support  meteorological  and  hydrological  service  functions. 

The  NWS  has  demonstrated  repeatedly  that  AFOS  can  be  modified 
efficiently,  effectively,  and  at  reasonable  cost,  and  based  on 
its  experience  expects  that  this  will  remain  the  case  for  many 
more  years.  Why  GAO  argues  that  AFOS  equipment  does  not  meet 
NWS  requirements,  has  limited  capacity,  and  is  expensive  to 
repair  in  light  of  the  actual  maintenance  record  is  puzzling. 

AFOS  Operating  System 

The  GAO  contends  that  the  computer's  operating  system  is 
inadequate,  unsuitable  for  the  purpose  intended,  and  must  be 
replaced.  The  AFOS  computers'  operating  system  is  software 
written  by  the  computer  manufacturer.  Computer  operating 
systems  being  sold  today  are  intended  to  serve  the  widest 
possible  spectrum  of  customer  needs.  In  developing  their 
operating  systems,  vendors  trade  off  between  highly  specialized 
and  efficient  software  systems  for  more  generalized,  and  in  many 
instances,  less  efficient  capabilities  that  serve  a  wider  market 
and  thus  yield  a  greater  return  on  investment.  The  AFOS 
operating  system  is  similiar  to  others  on  the  market  in  this 
respect.  The  NWS  has  modified  the  AFOS  operating  system  to 
enhance  its  efficiency  and  data-handling  capability  in  response 
to  specific  NWS  requirements  with  a  resultant  improvement  of  the 
performance  of  the  operating  system. 

The  computer's  operating  system  oversees  and  manages  most  of 
the  functions  of  the  computer.  Within  the  architecture  of  this 
operating  system,  a  user  generates  and  operates  his  own  computer 
programs  suited  to  his  specific  requirements  and  in  accordance 
with  the  rules  and  constraints  of  the  operating  system.  The 
AFOS  programs  contain  more  than  500,000  lines  of  computer  code 
that  satisfy  complex  requirements  and  perform  sophisticated 
functions  as  discussed  in  Chapter  4. 

The  operating  system  satisfies  the  basic  requirements  of  the 
AFOS  program.  Naturally  with  its  considerable  experience  in 
using  the  AFOS  operating  system,  i.e.,  hindsight,  it's  possible 
for  the  NWS  to  speculate  on  now  an  improved  operating  system 
might  be  developed,  but  there  are  no  operating  systems  on  the 
market  today  specifically  designed  to  satisfy  AFOS  requirements. 
The  NWS  did  attempt  to  obtain,  in  the  initial  hardware  contract, 
an  operating  system  built  specifically  for  AFOS  needs.  This 
path  had  to  be  abandoned  in  ±976  because  the  company  could  not 
build  and  deliver  this  hardware  on  the  time  schedule  prescribed 
in  the  contract.  The  NWS  has  chosen  the  most  reasonable  path 
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available  to  it  by  acquiring  an  off-the-shelf  operating  system 
that  had  proven  performance  and  then  specifically  tailoring  it 
to  the  needs  of  the  AFOS  program. 

Hardware  Capacity 

In  light  of  its  repeateo  and  extensive  tests  of  the 
performance  of  the  system,  the  NWS  does  not  agree  with  the  GAO’s 
comments  on  insufficient  hardware  capacity.  The  system  does  the 
job  originally  intended.  That  is  not  to  say  that  the  NWS  cannot 
improve  the  performance  or  capability  of  the  system,  it  can  and 
is  prepared  to  do  so  in  the  future. 

As  described  in  Chapter  l  of  these  comments,  the  AFOS 
system's  back-up  and  recovery  capability  is  in  place  and  is 
satisfactorily  performing  in  actual  field  use.  System  backups 
occur  on  multiple  levels;  eacn  with  its  own  specific  limitation 
on  performance  of  the  system.  Not  surprisingly,  practical 
experience  with  the  back-up  modes  of  AFOS  has  identified  areas 
to  strengthen.  The  NWS  intends  to  do  so  in  the  future. 

Hardware  Replacement 

Contrary  to  the  assertion  of  the  GAO,  the  AFOS  system  was 
not  intended  to  satisfy  all  future  requirements  of  the  NWS. 

This  is  in  accordance  with  the  Federal  ADP  procurement  policies. 
In  general,  new  generations  of  computer  systems  have  been 
introduced  every  5  to  7  years  that  prove  to  be  more  efficient 
and  effective  with  greater  capabilities  and  at  a  lesser  cost 
than  previous  ones.  The  presence  of  improved  generations  of 
equipment  in  itself  is  insufficient  reason  to  replace  systems 
if  they  are  still  economically  supportable  and  effective  tools 
in  support  of  NWS  operations. 

NWS  policy  in  regard  to  acquisition  of  equipment  systems 
incorporates  accepted  principles  of  economic  life  planning. 

Each  system  is  intended  to  satisfy  a  specific  set  of  agency 
requirements  and  not  be  so  generalized  (and  more  costly  and  less 
effective)  as  to  satisfy  new  and  different  changing  requirements 
over  the  years.  This  appears  to  be  a  point  of  philosophical 
disagreement  with  the  GAO.  As»  new  requirements  are  developed 
the  NWS  attempts  to  utilize  tne  latest  technologies  to  satisfy 
them.  Supporting  systems  maintenance  programs  are  designed  to 
provide  cost  effective  and  highly  reliable  systems  performance 
and  availability.  Subsequent  equipment  replacement  programs 
are  generated  as  a  result  of  closely  managing  the  costs  and 
effectiveness  of  systems  maintenance  and  NWS  operation 
requirements. 


i 


I 


APPENDIX  II 


APPENDIX  II 


CONCLUSIONS 


NOAA  does  not  agree  witn  the  GAO  conclusion  that  the  design 
and  capability  of  the  AFOS  hardware  and  operating  system  are 
inadequate.  The  AFOS  hardware  has  successfully  passed  repeated 
tests  and  evaluations  and  the  test  of  operational  use.  It  is 
satisfactory  for  AFOS  operations.  The  AFOS  computer's  operating 
system  has  been  tailored  to  tne  specific  application  of  AFOS 
required  for  successful  AFOS  performance. 

RECOMMENDATIONS 

NOAA  objects  to  the  GAO  recommendation  that  AFOS  hardware 
be  replaced  as  part  of  a  complete  system  redesign.  It's  NOAA's 
position  that  to  replace  satisfactory  operational  equipment  and 
discard  a  massive  logistic  and  repair  system  capable  of  meeting 
essential  NWS  requirements  for  support  of  AFOS  operations  is 
not  responsible  use  of  public  funds  and  human  resources. 
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CHAPTER  7 
FUTURE  OF  AFOS 


Tne  GAO  recommends  that  further  development  and  use  of  AFOS 
be  abandoned;  that  resources  be  redirected  to  planning  and 
developing  a  new  system  to  replace  AFOS  and  the  current  system; 
and  that  NWS  should  rely  solely  on  pre-AFOS  communication 
systems  until  the  new  system  is  developed.  NOAA  believes  that 
the  existing  AFOS  system  should  be  implemented  nationally  during 
FY  1982  as  the  primary  weather  information  system  supporting  NWS 
field  observations;  that  pre-AFOS  communications  be  used  as  a 
bacKup  to  the  AFOS  communications  subsystem,  and  to  support 
field  offices  not  equipped  with  AFOS,  until  the  end  of  FY  1984; 
and  that  a  next  generation  system  incorporating  advanced 
capabilities  never  intended  for  tne  initial  AFOS  system  should 
be  developed  for  implementation  by  the  end  of  the  decade. 

The  GAO  recommendations  propose  a  radical  departure  from 
the  plans  developed  and  substantially  fulfilled  by  AFOS  after  a 
major  investment  of  talent  and  at  least  $90M  over  a  period  of 
seven  years.  The  payoff  from  this  investment  would  be 
essentially  lost.  NOAA  believes  that  GAO  must  assume  a  neavy 
burden  of  justification  for  sucn  a  drastic  action.  The 
preceding  chapters  of  these  comments  have  systematically  shown 
that  tne  basic  conclusions  supporting  this  recommendation  do 
not  correspond  with  current  observable  facts,  are  logically 
inadequate  and  incomplete,  and  involve  inappropriate  and 
incorrect  judgments  about  the  requirements  of  the  NWS  and  the 
value  of  AFOS  to  NWS  operations.  Further,  NOAA  objects  to 
numerous  misattr ibutions  to  the  NWS  and  its  Director,  especially 
in  Chapter  7  of  the  draft  GAO  report,  in  the  form  botn  of 
inaccurate  paraphasing  and  selective  attributions  seriously  out 
of  context. 

In  order  to  provide  coherent  and  systematic  comments,  NOAA 
has  not  responded  in  each  case  to  repetitious  statements  and 
arguments  j.n  the  GAO  report  with  which  NOAA  disagrees.  The 
absence  of  an  explicit  rejection  by  NOAA  of  a  particular 
sentence  or  paragraph  must  not  be  ta Ken  as  tacit  acceptance  by 
NOAA.  However,  to  assist  in  sharpening  the  contrast  in  NOAA's 
and  GAO's  assessments  and  their  respective  foundations,  we  have 
attempted  to  assemble  a  more  concise  and  less  redundant 
comparison  in  this  section.  We  recognize  that  tnis  may  result 
in  some  loss  of  immediate  context  of  GAO's  arguments,  but  we 
have  made  an  honest  attempt  to  reflect  them  accurately,  and  the 
full  text  of  the  GAO  report  is  there  to  preserve  the  original 
context. 
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The  following  conclusions  stated  explicitly  by  GAO  (often 
with  more  or  different  words;  seem  to  form  the  central 
foundation  of  the  GAO's  main  recommendations: 

1.  AFOS  has  major  problems  in  its  software, 
hardware  and  telecommunications. 

2.  AFOS  is  not  sufficiently  stable  and  reliable 
to  use  as  a  primary  system. 

3.  AFOS  has  inadequate  backup  capabilities,  and 
the  most  critical  ones  in  the  original  design 
have  been  eliminated. 

4.  Improvement  of  AFOS  is  risky,  costly  and  im¬ 
practical  due  to  fundamental  design  flaws. 

5.  AFOS  is  too  expensive  to  operate  and  maintain. 

6.  AFOS  capabilities  do  not  meet  original  re¬ 
quirements,  and  a  complete  redesign  and  re¬ 
development  is  necessary  to  meet  them. 

7.  AFOS  cannot  meet  future  requirements. 

8.  The  value  of  AFOS  is  negligible. 

9.  Not  operating  AFOS  would  save  $116M  over 
the  next  8  years. 

10.  The  only  costs  associated  with  reverting  to 
pre-AFOS  operations  is  the  current  cost  of 
using  FAA's  teletypewriter  circuits. 

11.  NWS  can  operate  satisfactorily  and  meet  service 
requirements  for  the  rest  of  the  decade  without 
AFOS. 

12.  NWS  has  insufficient  staff  and  capability  to 
both  operate  and  maintain  AFOS,  and  plan  and 
develop  an  advanced  system. 

In  addition,  some  key  implicit  assumptions  underlying 
GAO's  recommendations  need  to  be  considered: 

13.  The  resources  previously  available  for  AFOS 
can  be  diverted  to  developing  a  new  system, 
would  be  sufficient  for  the  new  system,  and 
be  available  in  the  required  time  phasing. 
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14.  A  new  system  and  its  development  would  not  en¬ 
counter  technical,  cost,  schedule  or  transitional 
problems  as  serious  as  those  of  AFOS,  provided 
recommended  management  procedures  are  used  and 
development  is  done  by  contractors. 

In  contrast,  the  corresponding  points  of  NOAA's  assessment 
are  summarized  as  follows: 

1.  No  major  problems  in  AFOS  software,  hardware,  or 
telecommunications  remain  that  affect  its 
essential  functional  capabilities. 

2.  AFOS  is  sufficiently  stable  and  reliable  to  use 
as  the  primary  system;  the  existing  AFOS  system 
is  now  serving  as  the  primary  system  in  two  of 
the  four  contiguous  NWS  regions;  and  operational 
adequacy  will  be  demonstrated  during  August  and 
September  1981. 

3.  The  most  important  backup  capabilities  designed 
into  AFOS  have  been  implemented,  and  the  pre-AFOS 
communications  arrangements  will  be  retained  as 
an  added  backup  through  FY  1984. 

4.  Improvements  in  AFOS  have  been  incorporated  suc¬ 
cessfully  during  the  last  year,  and  NWS  believes 
that  some  investment  in  selective  improvements 
(not  fundamental  redesign  and  development) , 
retaining  the  existing  hardware  and  most  of  the 
software,  will  further  improve  system  performance 
and  facilitate  future  software  maintenance  and 
enhancement . 

5.  The  cost  of  operating  and  maintaining  AFOS  is 
close  to  the  projections  made  in  1976  when 
adjusted  for  the  effects  of  inflation  on  salaries 
and  spare  parts. 

6.  The  existing  AFOS  system  meets  all  significant 
original  requirements  except  for  early  witn- 
drawal  of  teletypewriter  and  facsimile  cap¬ 
abilities,  retained  as  backup,  and  substantially 
exceeds  original  expections  in  some  areas  such 
as  forecaster  assistance. 

7.  The  initial  AFOS  system  was  never  intended  or 
designed  to  meet  all  future  requirements,  and 
particularly  nut  those  cited  by  GAO. 
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8.  AFOS  is  extremely  valuable  to  NWS  operations;  it 
significantly  speeds  up  warnings  and  forecasts 
as  planned,  saves  large  numbers  of  staff-hours 
annually  through  automation  of  routine  tasks, 
liberates  time  and  resources  for  service 
improvements,  and  establishes  a  mode  of  operation 
amenable  to  future  improvements. 

9.  GAO's  estimate  that  $116M  could  be  saved  by  not 
operating  AFOS  for  8  years  does  not  consider 
important  costs  of  trying  to  turn  back  to 
operations  witnout  AFOS,  and  totally  ignores  the 
major  capabilities  and  efficiencies  that  would  be 
lost . 

10.  Reverting  to  pre-AFOS  systems  involves  much  more 
than  retaining  connection  to  FAA's  teletypewriter 
circuits,  including  refurbishing  or  replacing 
aging  equipment  of  various  types,  substantially 
increasing  field  personnel,  and  making  other 
potentially  expensive  adjustments. 

11.  Even  current  services  could  not  be  maintained 
using  pre-AFOS  systems  without  a  major 
investment  and  upheaval  in  NWS,  if  at  all. 

The  pre-AFOS  systems  do  not  meet  even  1974  NWS 
requirements;  that  is  why  the  AFOS  program  was 
initiated  to  provide  many  capabilities  not  in¬ 
cluded  at  all  in  the  previous  systems. 

12.  Totally  different  kinds  of  personnel  are  involved 
in  operating  AFOS  and  in  developing  a  new  system; 
NWS  can  accomplish  both  concurrently. 

13.  The  cost  of  a  new  system  meeting  all  present  and 
future  requirements  recommended  by  GAO  is 
unknown,  and  GAO's  estimate  of  $125  to  $150 
million  is  not  based  on  any  substantive  analysis 
or  knowledge  of  the  requirements;  and  the 
assumption  that  more  than  $100M,  now  planned  for 
expenditure  in  small  pieces  over  eight  years, 
can  be  gathered  up  for  a  procurement  is  at 

best  doubtful. 

14.  The  development  of  large,  complex  systems  that 
break  frontiers  should  be  expected  to  encounter 
some  problems;  NOAA's  experience  shows  that  pro¬ 
ceeding  in  a  series  of  ambitious  but  prudent 
steps  results  in  fewer  problems  than  making  a 
single  gianc  leap  to  a  new  system  incorporating 
even  more  novel  features  than  AFOS,  as  proposed 
by  GAO. 
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As  indicated  by  the  length  and  detail  of  these  comments, 
there  are  many  conclusions  by  the  GAO  that  directly  contradict 
NOAA's  assessment.  Much  of  the  contradiction  on  those  aspects 
related  to  sytem  performance  could  be  explained  by  the  fact 
that  GAO  auditors  made  their  observations  of  the  system  during 
an  earlier  period  when  development  and  testing  were  in  progress, 
and  many  technical  problems  were  evident.  We  can  only  assume 
that  these  difficulties  appeared  more  fundamental  and  fatal  to 
the  auditors  than  proved  to  be  the  case. 

We  are  troubled  by  the  analysis  of  the  alternatives  involved 
in  GAO's  recommendation  to  abandon  AFOS.  The  simplistic 
assertion  that  $116M  can  be  saved  over  the  next  eight  years  by 
terminating  AFOS  is  made  without  a  thoughtful  consideration  of 
the  relative  costs,  advantages  and  disadvantages  of  operation 
with  and  without  AFOS.  That  the  sole  cost  of  operations 
without  AFOS  is  considered  Dy  GAO  to  be  NWS'  cost  of  using  the 
FAA's  teletypewriter  circuits  (see  table  on  page  14),  while  the 
comparative  cost  of  operating  AFOS  includes  all  maintenance  and 
support  personnel,  facilities,  logistics,  communications 
documentation  and  even  millions  of  dollars  of  NOAA  overhead,  is 
not  logical.  That  the  difference  in  tne  functional 
capabilities  of  AFOS  and  teletypewriters  were  ignored  is  almost 
incredible . 

The  preceding  sections  or  these  comments  present  the  bases 
for  NOAA's  disagreements  witu  GAO's  conclusions  and 
recommendations.  These  bases  are  qualitatively  different  from 
those  used  by  GAO.  NOAA  bases  its  conclusions  upon  thorough, 
up-to-date,  first-hand  experience  in  testing  and  using  AFOS  in 
support  of  weather  service  operations;  upon  deep  understanding 
of  the  requirements  the  system  must  satisfy  and  tneir  relative 
importance  and  urgency  to  weather  service  operations;  and  upon 
its  knowledge  of  the  importance  of  various  current  and  needed 
weather  services  derived  from  more  than  a  century  of  working 
directly  with  NWS  users. 
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